[jira] [Created] (HDDS-721) NullPointerException thrown while trying to read a file when datanode restarted
Nilotpal Nandi created HDDS-721: --- Summary: NullPointerException thrown while trying to read a file when datanode restarted Key: HDDS-721 URL: https://issues.apache.org/jira/browse/HDDS-721 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Affects Versions: 0.3.0 Reporter: Nilotpal Nandi steps taken : --- # Put few files and directories using ozonefs # stopped all services of cluster. # started the scm, om and then datanodes. While datanodes were starting up, tried to read a file. Null pointer Exception was thrown. {noformat} [root@ctr-e138-1518143905142-53-01-03 ~]# /root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/bin/ozone fs -ls -R / 2018-10-24 04:48:00,703 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable drwxrwxrwx - root root 0 2018-10-24 04:12 /testdir1 -rw-rw-rw- 1 root root 5368709120 1970-02-25 15:29 /testdir1/5GB -rw-rw-rw- 1 root root 4798 1970-02-25 15:22 /testdir1/passwd drwxrwxrwx - root root 0 2018-10-24 04:46 /testdir3 [root@ctr-e138-1518143905142-53-01-03 ~]# /root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/bin/ozone fs -cat o3fs://fs-bucket.fs-volume/testdir1/passwd 2018-10-24 04:49:24,955 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable cat: Exception getting XceiverClient: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.NullPointerException{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-722) ozone datanodes failed to start on few nodes
Nilotpal Nandi created HDDS-722: --- Summary: ozone datanodes failed to start on few nodes Key: HDDS-722 URL: https://issues.apache.org/jira/browse/HDDS-722 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Affects Versions: 0.3.0 Reporter: Nilotpal Nandi steps taken : -- # put few keys using ozonefs. # stopped all services of the cluster. # started om and scm. # After sometime , started datanodes. All datanodes failed to start . Out of 12 datanodes, 4 datanodes failed to start. Here is the datanode log snippet : {noformat} 2018-10-24 04:49:30,594 ERROR org.apache.ratis.server.impl.StateMachineUpdater: Terminating with exit status 2: StateMachineUpdater-9524f4e2-9031-4852-ab7c-11c2da3460db: the StateMachineUpdater hits Throwable org.apache.ratis.server.storage.RaftLogIOException: java.io.IOException: Premature EOF from inputStream at org.apache.ratis.server.storage.LogSegment.loadCache(LogSegment.java:299) at org.apache.ratis.server.storage.SegmentedRaftLog.get(SegmentedRaftLog.java:192) at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:142) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Premature EOF from inputStream at org.apache.ratis.util.IOUtils.readFully(IOUtils.java:100) at org.apache.ratis.server.storage.LogReader.decodeEntry(LogReader.java:250) at org.apache.ratis.server.storage.LogReader.readEntry(LogReader.java:155) at org.apache.ratis.server.storage.LogInputStream.nextEntry(LogInputStream.java:128) at org.apache.ratis.server.storage.LogSegment.readSegmentFile(LogSegment.java:110) at org.apache.ratis.server.storage.LogSegment.access$400(LogSegment.java:43) at org.apache.ratis.server.storage.LogSegment$LogEntryLoader.load(LogSegment.java:167) at org.apache.ratis.server.storage.LogSegment$LogEntryLoader.load(LogSegment.java:161) at org.apache.ratis.server.storage.LogSegment.loadCache(LogSegment.java:295) ... 3 more 2018-10-24 04:49:30,598 INFO org.apache.hadoop.ozone.HddsDatanodeService: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down HddsDatanodeService at ctr-e138-1518143905142-541661-01-03.hwx.site/172.27.57.0 / 2018-10-24 04:49:30,598 WARN org.apache.hadoop.fs.CachingGetSpaceUsed: Thread Interrupted waiting to refresh disk information: sleep interrupted {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-723) CloseContainerCommandHandler throwing NullPointerException
Nilotpal Nandi created HDDS-723: --- Summary: CloseContainerCommandHandler throwing NullPointerException Key: HDDS-723 URL: https://issues.apache.org/jira/browse/HDDS-723 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Affects Versions: 0.3.0 Reporter: Nilotpal Nandi Seeing NullPointerException error while CloseContainerCommandHandler is trying to close container. {noformat} 2018-10-24 04:22:04,699 INFO org.apache.ratis.server.storage.RaftLogWorker: 8a61160b-8985-412e-9f25-9e65ceafa824-RaftLogWorker got closed and hit exception java.io.IOException: java.lang.InterruptedException at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:51) at org.apache.ratis.server.storage.RaftLogWorker.flushWrites(RaftLogWorker.java:232) at org.apache.ratis.server.storage.RaftLogWorker.access$600(RaftLogWorker.java:51) at org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:309) at org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:179) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.InterruptedException at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.ratis.server.storage.RaftLogWorker.flushWrites(RaftLogWorker.java:230) ... 4 more 2018-10-24 04:22:04,712 INFO org.apache.ratis.server.storage.RaftLogWorker: 8a61160b-8985-412e-9f25-9e65ceafa824-RaftLogWorker close() 2018-10-24 04:22:31,293 ERROR org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler: Can't close container 18 java.lang.NullPointerException at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:78) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:93) at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$1(DatanodeStateMachine.java:381) at java.lang.Thread.run(Thread.java:745) 2018-10-24 04:22:31,293 ERROR org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler: Can't close container 10 java.lang.NullPointerException at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:78) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:93) at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$1(DatanodeStateMachine.java:381) at java.lang.Thread.run(Thread.java:745) 2018-10-24 04:22:31,293 ERROR org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler: Can't close container 14 java.lang.NullPointerException at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:78) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:93) at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$1(DatanodeStateMachine.java:381) at java.lang.Thread.run(Thread.java:745){noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-725) Exception thrown in loop while trying to write a file in ozonefs
Nilotpal Nandi created HDDS-725: --- Summary: Exception thrown in loop while trying to write a file in ozonefs Key: HDDS-725 URL: https://issues.apache.org/jira/browse/HDDS-725 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Affects Versions: 0.3.0 Environment: Ran the following command : ozone fs -put 2GB /testdir5/ Exceptions are thrown continuously in loop. Please note that there are 8 datanodes alive in the cluster. {noformat} root@ctr-e138-1518143905142-53-01-08 logs]# /root/allssh.sh 'jps -l | grep Datanode' Host::172.27.20.96 411564 org.apache.hadoop.ozone.HddsDatanodeService Host::172.27.20.91 472897 org.apache.hadoop.ozone.HddsDatanodeService Host::172.27.38.9 351139 org.apache.hadoop.ozone.HddsDatanodeService Host::172.27.24.90 314304 org.apache.hadoop.ozone.HddsDatanodeService Host::172.27.15.139 324820 org.apache.hadoop.ozone.HddsDatanodeService Host::172.27.10.199 Host::172.27.15.131 Host::172.27.57.0 Host::172.27.23.139 627053 org.apache.hadoop.ozone.HddsDatanodeService Host::172.27.68.65 557443 org.apache.hadoop.ozone.HddsDatanodeService Host::172.27.19.74 Host::172.27.85.64 508121 org.apache.hadoop.ozone.HddsDatanodeService{noformat} {noformat} 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: Election REJECTED; received 0 response(s) [] and 2 exception(s); 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:t16296, leader=null, voted=7c3b2fb1-cf16-4e5f-94dc-8a089492ad57, raftlog=[(t:37, i:271)], conf=271: [7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:172.27.85.64:9858, 86f9e313-ae49-4675-95d7-27856641aee1:172.27.15.131:9858, 9524f4e2-9031-4852-ab7c-11c2da3460db:172.27.57.0:9858], old=null 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RaftServerImpl: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from CANDIDATE to FOLLOWER at term 16296 for changeToFollower 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown LeaderElection 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start FollowerState 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.FollowerState: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes to CANDIDATE, lastRpcTime:1078, electionTimeout:1078ms 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown FollowerState 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RaftServerImpl: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from FOLLOWER to CANDIDATE at term 16296 for changeToCandidate 2018-10-24 09:49:48,172 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start LeaderElection 2018-10-24 09:49:48,173 INFO org.apache.ratis.server.impl.LeaderElection: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: begin an election in Term 16297 2018-10-24 09:49:48,174 INFO org.apache.ratis.server.impl.LeaderElection: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 got exception when requesting votes: {} java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214) at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146) at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102) Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222) at org.apache
[jira] [Created] (HDDS-727) ozone.log is not getting created in logs directory
Nilotpal Nandi created HDDS-727: --- Summary: ozone.log is not getting created in logs directory Key: HDDS-727 URL: https://issues.apache.org/jira/browse/HDDS-727 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Affects Versions: 0.3.0 Reporter: Nilotpal Nandi ozone.log is no more present in logs directory of datanodes. Need to be added back. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-736) scm UI conf page throwing error 'HTTP ERROR 500'
Nilotpal Nandi created HDDS-736: --- Summary: scm UI conf page throwing error 'HTTP ERROR 500' Key: HDDS-736 URL: https://issues.apache.org/jira/browse/HDDS-736 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Affects Versions: 0.3.0 Reporter: Nilotpal Nandi Attachments: error_scm_ui.png scm UI conf page throwing error 'HTTP ERROR 500' . Please refer the screenshot. !error_scm_ui.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-740) datanode failed to start due 'Corrupted log header'
Nilotpal Nandi created HDDS-740: --- Summary: datanode failed to start due 'Corrupted log header' Key: HDDS-740 URL: https://issues.apache.org/jira/browse/HDDS-740 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.3.0 Reporter: Nilotpal Nandi steps taken : -- # wrote 5GB files through ozonefs # stopped datanodes, scm , om. # started all services. # Tried to read the file. One of the datanodes failed to start. Throwing "java.lang.IllegalStateException: Corrupted log header" {noformat} 2018-10-26 10:26:01,317 ERROR org.apache.ratis.server.storage.LogInputStream: caught exception initializing log_inprogress_293 java.lang.IllegalStateException: Corrupted log header: ^@^@^@^@^@^@^@^@ at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60) at org.apache.ratis.server.storage.LogInputStream.init(LogInputStream.java:93) at org.apache.ratis.server.storage.LogInputStream.nextEntry(LogInputStream.java:120) at org.apache.ratis.server.storage.LogSegment.readSegmentFile(LogSegment.java:111) at org.apache.ratis.server.storage.LogSegment.loadSegment(LogSegment.java:133) at org.apache.ratis.server.storage.RaftLogCache.loadSegment(RaftLogCache.java:110) at org.apache.ratis.server.storage.SegmentedRaftLog.loadLogSegments(SegmentedRaftLog.java:151) at org.apache.ratis.server.storage.SegmentedRaftLog.open(SegmentedRaftLog.java:120) at org.apache.ratis.server.impl.ServerState.initLog(ServerState.java:191) at org.apache.ratis.server.impl.ServerState.(ServerState.java:114) at org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:106) at org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:196) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) 2018-10-26 10:26:03,671 INFO org.apache.hadoop.ozone.web.netty.ObjectStoreRestHttpServer: Listening HDDS REST traffic on /0.0.0.0:9880 2018-10-26 10:26:03,672 INFO org.apache.hadoop.ozone.HddsDatanodeService: Started plug-in org.apache.hadoop.ozone.web.OzoneHddsDatanodeService@1e411d81 2018-10-26 10:26:03,676 INFO org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer: Attempting to start container services. 2018-10-26 10:26:03,676 INFO org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis: Starting XceiverServerRatis 0d7f5327-df16-40fe-ac88-7ed06e76a20f at port 9858 2018-10-26 10:26:03,702 ERROR org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine: Unable to start the DatanodeState Machine java.io.IOException: java.lang.IllegalStateException: Corrupted log header: ^@^@^@^@^@^@^@^@ at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:51) at org.apache.ratis.server.storage.LogInputStream.nextEntry(LogInputStream.java:123) at org.apache.ratis.server.storage.LogSegment.readSegmentFile(LogSegment.java:111) at org.apache.ratis.server.storage.LogSegment.loadSegment(LogSegment.java:133) at org.apache.ratis.server.storage.RaftLogCache.loadSegment(RaftLogCache.java:110) at org.apache.ratis.server.storage.SegmentedRaftLog.loadLogSegments(SegmentedRaftLog.java:151) at org.apache.ratis.server.storage.SegmentedRaftLog.open(SegmentedRaftLog.java:120) at org.apache.ratis.server.impl.ServerState.initLog(ServerState.java:191) at org.apache.ratis.server.impl.ServerState.(ServerState.java:114) at org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:106) at org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$2(RaftServerProxy.java:196) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) Caused by: java.lang.IllegalStateException: Corrupted log header: ^@^@^@^@^@^@^@^@ at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60) at org.apache.ratis.server.storage.LogInputStream.init(LogInputStream.java:93) at org.apache.ratis.server.storage.LogInputStream.nextEntry(LogInputStream.java:120) ... 15 more 2018-10-26 10:26:03,703 ERROR org.apache.hadoop.ozone.HddsDatanodeService: Except
[jira] [Created] (HDDS-741) all pipelines are in 'CLOSING' state after service restart
Nilotpal Nandi created HDDS-741: --- Summary: all pipelines are in 'CLOSING' state after service restart Key: HDDS-741 URL: https://issues.apache.org/jira/browse/HDDS-741 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Affects Versions: 0.3.0 Reporter: Nilotpal Nandi all pipelines are in closing state after service restart. None of the writes are working after restart. The cluster contains 11 live datanodes. ** {noformat} [ { "nodeType": "OM", "hostname": "ctr-e138-1518143905142-53-01-08.hwx.site", "ports": { "RPC": 9889, "HTTP": 9874 } }, { "nodeType": "SCM", "hostname": "ctr-e138-1518143905142-53-01-03.hwx.site", "ports": { "RPC": 9860 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-541661-01-03.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-541661-01-07.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-53-01-03.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-541661-01-04.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-53-01-04.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-53-01-08.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-541661-01-02.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-53-01-05.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-541661-01-06.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-53-01-07.hwx.site", "ports": { "HTTP": 9880 } }, { "nodeType": "DATANODE", "hostname": "ctr-e138-1518143905142-53-01-06.hwx.site", "ports": { "HTTP": 9880 } } ]{noformat} error thrown while write : {noformat} [root@ctr-e138-1518143905142-541661-01-07 test_files]# ozone fs -put /etc/passwd /testdir5/ 2018-10-26 12:09:43,822 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-10-26 12:09:47,882 ERROR io.ChunkGroupOutputStream: Try to allocate more blocks for write failed, already allocated 0 blocks for this write. put: Allocate block failed, error:INTERNAL_ERROR{noformat} pipelines in the cluster : {noformat} [root@ctr-e138-1518143905142-541661-01-07 test_files]# ozone scmcli listPipelines Pipeline[ Id: 29b68cc2-2d18-4db0-a11a-587ae4abc715, Nodes: e3d89961-fe38-4ed0-8a32-cd1849c58e0c{ip: 172.27.20.96, host: ctr-e138-1518143905142-53-01-08.hwx.site}b33a30d9-f1e2-448e-aabb-61a970445cea{ip: 172.27.85.64, host: ctr-e138-1518143905142-541661-01-07.hwx.site}, Type:RATIS, Factor:THREE, State:CLOSING] Pipeline[ Id: 05061f87-4c68-443b-ae27-984da2d0a2cd, Nodes: dc002a73-fc63-4e76-be3e-3c6d16ede5f6{ip: 172.27.38.9, host: ctr-e138-1518143905142-53-01-04.hwx.site}4e6bd2a2-6802-4e67-9710-612a2cdb9dc1{ip: 172.27.24.90, host: ctr-e138-1518143905142-53-01-05.hwx.site}be3f0db4-3a19-44a5-bd6e-0da47d2ed92e{ip: 172.27.20.91, host: ctr-e138-1518143905142-53-01-03.hwx.site}, Type:RATIS, Factor:THREE, State:CLOSING] Pipeline[ Id: 80893f87-5e73-49a2-8f38-2adb2b13140a, Nodes: 63833540-bf93-410c-b081-243a56f93c88{ip: 172.27.10.199, host: ctr-e138-1518143905142-53-01-07.hwx.site}6e8b7129-8615-45fe-81e0-848a2e0ba520{ip: 172.27.15.139, host: ctr-e138-1518143905142-53-01-06.hwx.site}aab1f2e5-1cf0-430d-b1bf-04be8630a8ee{ip: 172.27.57.0, host: ctr-e138-1518143905142-541661-01-03.hwx.site}, Type:RATIS, Factor:THREE, State:CLOSING] Pipeline[ Id: f0a14cb9-d37a-4c7c-b3e6-b7e3830dfd5f, Nodes: 61e271bf-68ad-435e-8a6e-582be90ebb6f{ip: 172.27.19.74, host: ctr-e138-1518143905142-541661-01-06.hwx.site}3622352c-b136-4c74-b952-34e938cbda94{ip: 172.27.15.131, host: ctr-e138-1518143905142-541661-01-02.hwx.site}cb2b1e95-e803-48d3-bdf2-bf878cae62cf{ip: 172.27.23.139, host: ctr-e138-1518143905142-541661-01-04.hwx.site}, Type:RATIS, Factor:THREE, State:CLOSING] {noformat} datanode.log : --- {noformat} 2018-10-26 12:17:23,697 INFO org.apache.ratis.server.impl.LeaderElection: e3d89961-fe38-4ed0-8a32-cd1849c58e0c: Election REJECTED; received 1 response(s) [e3d89961-fe38-4ed0-8a32-cd1849c58e0c<-b33a30d9-f1e2-448e-aabb-61a970445cea#0:FAIL-t1019] and 1 exception(s); e3d89961-fe38-4ed0-8a32-cd1849c58e0c:t1019, leader=null, voted=e3d89961-fe38-4ed0-8a32-cd1849c58e0c, raftlog=e3d89961-fe38-4ed0-8a32-cd1849c58e0c-Segmen
[jira] [Created] (HDDS-768) writeStateMachineData times out
Nilotpal Nandi created HDDS-768: --- Summary: writeStateMachineData times out Key: HDDS-768 URL: https://issues.apache.org/jira/browse/HDDS-768 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.3.0 Reporter: Nilotpal Nandi datanode stopped due to following error : datanode.log {noformat} 2018-10-31 09:12:04,517 INFO org.apache.ratis.server.impl.RaftServerImpl: 9fab9937-fbcd-4196-8014-cb165045724b: set configuration 169: [9fab9937-fbcd-4196-8014-cb165045724b:172.27.15.131:9858, ce0084c2-97cd-4c97-9378-e5175daad18b:172.27.15.139:9858, f0291cb4-7a48-456a-847f-9f91a12aa850:172.27.38.9:9858], old=null at 169 2018-10-31 09:12:22,187 ERROR org.apache.ratis.server.storage.RaftLogWorker: Terminating with exit status 1: 9fab9937-fbcd-4196-8014-cb165045724b-RaftLogWorker failed. org.apache.ratis.protocol.TimeoutIOException: Timeout: WriteLog:182: (t:10, i:182), STATEMACHINELOGENTRY, client-611073BBFA46, cid=127-writeStateMachineData at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:87) at org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:310) at org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:182) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:79) ... 3 more{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-769) temporary file ._COPYING_ is not deleted after put command failure
Nilotpal Nandi created HDDS-769: --- Summary: temporary file ._COPYING_ is not deleted after put command failure Key: HDDS-769 URL: https://issues.apache.org/jira/browse/HDDS-769 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Affects Versions: 0.3.0 Reporter: Nilotpal Nandi steps taken : - # stopped all datanodes. # Ran ozonefs put command. Command execution failed. {noformat} [root@ctr-e138-1518143905142-53-01-08 ~]# ozone fs -put /etc/passwd /testdir5/ 2018-10-31 08:42:12,711 [main] ERROR - Try to allocate more blocks for write failed, already allocated 0 blocks for this write. put: Allocate block failed, error:INTERNAL_ERROR{noformat} But , the temporary file was not deleted from OM. {noformat} [root@ctr-e138-1518143905142-53-01-03 logs]# ozone fs -ls /testdir5/passwd._COPYING_ -rw-rw-rw- 1 root root 0 2018-10-31 08:42 /testdir5/passwd._COPYING_{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-770) ozonefs client warning exception logs should not be displayed on console
Nilotpal Nandi created HDDS-770: --- Summary: ozonefs client warning exception logs should not be displayed on console Key: HDDS-770 URL: https://issues.apache.org/jira/browse/HDDS-770 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Affects Versions: 0.3.0 Reporter: Nilotpal Nandi steps taken : - # ran ozonefs cp command - "ozone fs -cp /testdir2/2GB /testdir2/2GB_111" # command execution was successful and file was successfully copied. But , the warning logs/exceptions are displayed on console : {noformat} [root@ctr-e138-1518143905142-53-01-03 ~]# ozone fs -cp /testdir2/2GB /testdir2/2GB_111 2018-10-31 09:12:35,052 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: GetBlock traceID: "b73d7d2d-232a-40d7-b0b6-478e3d40ed6a" containerID: 17 datanodeUuid: "ce0084c2-97cd-4c97-9378-e5175daad18b" getBlock { blockID { containerID: 17 localID: 100989077200109583 } blockCommitSequenceId: 60 } on datanode 9fab9937-fbcd-4196-8014-cb165045724b java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:167) at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:146) at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:105) at org.apache.hadoop.ozone.client.io.ChunkGroupInputStream.getFromOmKeyInfo(ChunkGroupInputStream.java:301) at org.apache.hadoop.ozone.client.rpc.RpcClient.getKey(RpcClient.java:493) at org.apache.hadoop.ozone.client.OzoneBucket.readKey(OzoneBucket.java:272) at org.apache.hadoop.fs.ozone.OzoneFileSystem.open(OzoneFileSystem.java:178) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:950) at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:341) at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277) at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262) at org.apache.hadoop.fs.shell.Command.processPathInternal(Command.java:367) at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331) at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:304) at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257) at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:286) at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270) at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120) at org.apache.hadoop.fs.shell.Command.run(Command.java:177) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:390) Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526) at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420) at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403) at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at org.apache.r
[jira] [Created] (HDDS-772) ratis retries infinitely and does not timeout when datanode goes down
Nilotpal Nandi created HDDS-772: --- Summary: ratis retries infinitely and does not timeout when datanode goes down Key: HDDS-772 URL: https://issues.apache.org/jira/browse/HDDS-772 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Affects Versions: 0.3.0 Reporter: Nilotpal Nandi steps taken : - # Ran ozonefs client operations. # Some of the datanodes were down. # client operations did not fail and are in waiting/hung state. reason: RATIS retries infinitely. datanode.log {noformat} 2018-10-31 11:13:28,423 WARN org.apache.ratis.grpc.server.GrpcServerProtocolService: 046351fe-bb76-4f86-b296-c682746981c4: Failed requestVote 54026017-a738-45f5-92f9-c50a0fc24a9f->046351fe-bb76-4f86-b296-c682746981c4#0 org.apache.ratis.protocol.GroupMismatchException: 046351fe-bb76-4f86-b296-c682746981c4: group-FF58136AA1BA not found. at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114) at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:257) at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:266) at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261) at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:428) at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54) at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707) at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-10-31 11:13:29,574 WARN org.apache.ratis.grpc.server.GrpcServerProtocolService: 046351fe-bb76-4f86-b296-c682746981c4: Failed requestVote 54026017-a738-45f5-92f9-c50a0fc24a9f->046351fe-bb76-4f86-b296-c682746981c4#0 org.apache.ratis.protocol.GroupMismatchException: 046351fe-bb76-4f86-b296-c682746981c4: group-FF58136AA1BA not found. at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114) at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:257) at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:266) at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261) at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:428) at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54) at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319) at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707) at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-10-31 11:13:30,772 WARN org.apache.ratis.grpc.server.GrpcServerProtocolService: 046351fe-bb76-4f86-b296-c682746981c4: Failed requestVote 54026017-a738-45f5-92f9-c50a0fc24a9f->046351fe-bb76-4f86-b296-c682746981c4#0 org.apache.ratis.protocol.GroupMismatchException: 046351fe-bb76-4f86-b296-c682746981c4: group-FF58136AA1BA not found. at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114) at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:257) at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:266) at org.apache.ratis.server
[jira] [Created] (HDDS-874) ozonefs write operation failed
Nilotpal Nandi created HDDS-874: --- Summary: ozonefs write operation failed Key: HDDS-874 URL: https://issues.apache.org/jira/browse/HDDS-874 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Filesystem Reporter: Nilotpal Nandi Here is the error thrown on console while trying to write a file to ozone filesystem {noformat} [root@ctr-e139-1542663976389-11261-01-05 test_files]# ozone fs -copyFromLocal 5GB / 2018-11-27 04:24:14,117 WARN scm.XceiverClientRatis: 3 way commit failed java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.apache.hadoop.hdds.scm.XceiverClientRatis.watchForCommit(XceiverClientRatis.java:162) at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.watchForCommit(ChunkOutputStream.java:303) at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.handleFullBuffer(ChunkOutputStream.java:289) at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.write(ChunkOutputStream.java:193) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.write(ChunkGroupOutputStream.java:695) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:292) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.write(ChunkGroupOutputStream.java:251) at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.write(OzoneFSOutputStream.java:47) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57) at java.io.DataOutputStream.write(DataOutputStream.java:107) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:96) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:68) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129) at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:485) at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407) at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342) at org.apache.hadoop.fs.shell.CopyCommands$CopyFromLocal.copyFile(CopyCommands.java:357) at org.apache.hadoop.fs.shell.CopyCommands$CopyFromLocal.copyFileToTarget(CopyCommands.java:365) at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277) at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262) at org.apache.hadoop.fs.shell.Command.processPathInternal(Command.java:367) at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331) at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:304) at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257) at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:286) at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270) at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228) at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:295) at org.apache.hadoop.fs.shell.CopyCommands$CopyFromLocal.processArguments(CopyCommands.java:385) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120) at org.apache.hadoop.fs.shell.Command.run(Command.java:177) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.fs.FsShell.main(FsShell.java:390) 2018-11-27 04:24:17,497 INFO scm.XceiverClientRatis: Could not commit 146 to all the nodes.Committed by majority. 2018-11-27 04:24:55,867 WARN scm.XceiverClientRatis: 3 way commit failed java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.apache.hadoop.hdds.scm.XceiverClientRatis.watchForCommit(XceiverClientRatis.java:162) at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.watchForCommit(ChunkOutputStream.java:303) at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.handleFullBuffer(ChunkOutputStream.java:289) at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.write(ChunkOutputStream.java:193) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.write(ChunkGroupOutputStream.java:695) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:292) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.write(ChunkGroupOutputStream.java:251) at org.apache.hadoop.fs.ozone.OzoneFSO
[jira] [Created] (HDDS-893) pipeline status is ALLOCATED in scmcli listPipelines command
Nilotpal Nandi created HDDS-893: --- Summary: pipeline status is ALLOCATED in scmcli listPipelines command Key: HDDS-893 URL: https://issues.apache.org/jira/browse/HDDS-893 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Reporter: Nilotpal Nandi Pipeline status cannot be allocated , It should be either OPEN or CLOSING. {noformat} [root@ctr-e139-1542663976389-11261-01-05 test_files]# ozone scmcli listPipelines Pipeline[ Id: 202f7208-6977-4f65-b070-c1e7e57cb2ed, Nodes: 06e074f7-67b4-4dde-8f20-a437ca60b7a1{ip: 172.27.20.97, host: ctr-e139-1542663976389-11261-01-07.hwx.site}c5bf9a9f-d471-4cef-aae4-61cb387ea9e3{ip: 172.27.79.145, host: ctr-e139-1542663976389-11261-01-06.hwx.site}96c18fe3-5520-4941-844b-ff7186a146a6{ip: 172.27.55.132, host: ctr-e139-1542663976389-11261-01-03.hwx.site}, Type:RATIS, Factor:THREE, State:ALLOCATED]{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-897) all healthy datanodes are not involved in pipeline allocation
Nilotpal Nandi created HDDS-897: --- Summary: all healthy datanodes are not involved in pipeline allocation Key: HDDS-897 URL: https://issues.apache.org/jira/browse/HDDS-897 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode, SCM Reporter: Nilotpal Nandi Attachments: active_datanode1.png, active_datanode2.png Please find the attached screenshot. There are 6 active datanodes , but there is only 1 pipeline created. All new writes are now done on a single pipeline even when there are 3 more free and active datanodes. !active_datanode2.png! !active_datanode1.png! {noformat} [root@ctr-e139-1542663976389-11261-01-07 meta]# ozone scmcli listPipelines Pipeline[ Id: 202f7208-6977-4f65-b070-c1e7e57cb2ed, Nodes: 06e074f7-67b4-4dde-8f20-a437ca60b7a1{ip: 172.27.20.97, host: ctr-e139-1542663976389-11261-01-07.hwx.site}c5bf9a9f-d471-4cef-aae4-61cb387ea9e3{ip: 172.27.79.145, host: ctr-e139-1542663976389-11261-01-06.hwx.site}96c18fe3-5520-4941-844b-ff7186a146a6{ip: 172.27.55.132, host: ctr-e139-1542663976389-11261-01-03.hwx.site}, Type:RATIS, Factor:THREE, State:ALLOCATED]{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-899) Include pipeline info and leader info in scm UI metric page
Nilotpal Nandi created HDDS-899: --- Summary: Include pipeline info and leader info in scm UI metric page Key: HDDS-899 URL: https://issues.apache.org/jira/browse/HDDS-899 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Nilotpal Nandi Open pipeline info and corresponding leader information should be included in SCM UI metric page ( HTTP port : 9876). These information would be helpful while debugging the issues live. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-900) scmcli info cannot find container if the container does not contain any key
Nilotpal Nandi created HDDS-900: --- Summary: scmcli info cannot find container if the container does not contain any key Key: HDDS-900 URL: https://issues.apache.org/jira/browse/HDDS-900 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Reporter: Nilotpal Nandi Please check the commands below. scmcli 'list' command shows the container but the 'info' command does not show it . {noformat} [root@ctr-e139-1542663976389-22144-01-02 test_files]# ozone scmcli list --start=19 { "state" : "OPEN", "replicationFactor" : "THREE", "replicationType" : "RATIS", "usedBytes" : 0, "numberOfKeys" : 0, "lastUsed" : 6719234213, "stateEnterTime" : 9829697449, "owner" : "16b9d1db-cfca-4d46-bdfa-7ee136ba626a", "containerID" : 20, "deleteTransactionId" : 0, "sequenceId" : 0, "open" : true }{noformat} [root@ctr-e139-1542663976389-22144-01-02 test_files]# ozone scmcli info 20 ContainerID 20 does not exist -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-904) RATIS group not found thrown on datanodes while leader election
Nilotpal Nandi created HDDS-904: --- Summary: RATIS group not found thrown on datanodes while leader election Key: HDDS-904 URL: https://issues.apache.org/jira/browse/HDDS-904 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode, SCM Reporter: Nilotpal Nandi Attachments: datanode_1.log, datanode_2.log, datanode_3.log, scm.log Following exception seen in datanode.log of one the docker nodes - {noformat} 2018-12-06 09:32:11 INFO LeaderElection:127 - 0e3aa95d-ab51-4b20-9bff-3f7bd7df0500: begin an election in Term 1 2018-12-06 09:32:12 INFO LeaderElection:46 - 0e3aa95d-ab51-4b20-9bff-3f7bd7df0500: Election TIMEOUT; received 0 response(s) [] and 0 exception(s); 0e3aa95d-ab51-4b20-9bff-3f7bd7df0500:t1, leader=null, voted=0e3aa95d-ab51-4b20-9bff-3f7bd7df0500, raftlog=0e3aa95d-ab51-4b20-9bff-3f7bd7df0500-SegmentedRaftLog:OPENED, conf=-1: [76153aab-4681-40b6-bc32-cc9ed5ef1daf:192.168.0.7:9858, 79ca7251-7514-4c53-968c-ade59d6df07b:192.168.0.6:9858, 0e3aa95d-ab51-4b20-9bff-3f7bd7df0500:192.168.0.4:9858], old=null 2018-12-06 09:32:13 INFO LeaderElection:127 - 0e3aa95d-ab51-4b20-9bff-3f7bd7df0500: begin an election in Term 2 2018-12-06 09:32:13 INFO LeaderElection:230 - 0e3aa95d-ab51-4b20-9bff-3f7bd7df0500 got exception when requesting votes: {} java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 76153aab-4681-40b6-bc32-cc9ed5ef1daf: group-41B8C34A6DE4 not found. at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214) at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146) at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102) Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: 76153aab-4681-40b6-bc32-cc9ed5ef1daf: group-41B8C34A6DE4 not found. at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222) at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203) at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132) at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:63) at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:150) at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-12-06 09:32:14 INFO LeaderElection:46 - 0e3aa95d-ab51-4b20-9bff-3f7bd7df0500: Election TIMEOUT; received 0 response(s) [] and 1 exception(s); 0e3aa95d-ab51-4b20-9bff-3f7bd7df0500:t2, leader=null, voted=0e3aa95d-ab51-4b20-9bff-3f7bd7df0500, raftlog=0e3aa95d-ab51-4b20-9bff-3f7bd7df0500-SegmentedRaftLog:OPENED, conf=-1: [76153aab-4681-40b6-bc32-cc9ed5ef1daf:192.168.0.7:9858, 79ca7251-7514-4c53-968c-ade59d6df07b:192.168.0.6:9858, 0e3aa95d-ab51-4b20-9bff-3f7bd7df0500:192.168.0.4:9858], old=null{noformat} cc - [~ljain] all logs attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-932) Add blockade Tests for Network partition
Nilotpal Nandi created HDDS-932: --- Summary: Add blockade Tests for Network partition Key: HDDS-932 URL: https://issues.apache.org/jira/browse/HDDS-932 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Affects Versions: 0.4.0 Reporter: Nilotpal Nandi Assignee: Nilotpal Nandi Fix For: 0.4.0 Blockade tests need to be added pertaining to network partition. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-934) freon run hung and did not terminate when run on non-functional pipeline
Nilotpal Nandi created HDDS-934: --- Summary: freon run hung and did not terminate when run on non-functional pipeline Key: HDDS-934 URL: https://issues.apache.org/jira/browse/HDDS-934 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi Attachments: jstack.txt steps taken: # created docker cluster with 3 datanodes where datanodes are not able to communicate with each other. # ran freon to write 1 key freon run threw following exception but it did not terminate. exception: -- {noformat} 2018-12-18 10:38:06 INFO RandomKeyGenerator:227 - Number of Threads: 10 2018-12-18 10:38:06 INFO RandomKeyGenerator:233 - Number of Volumes: 1. 2018-12-18 10:38:06 INFO RandomKeyGenerator:234 - Number of Buckets per Volume: 1. 2018-12-18 10:38:06 INFO RandomKeyGenerator:235 - Number of Keys per Bucket: 1. 2018-12-18 10:38:06 INFO RandomKeyGenerator:236 - Key size: 10240 bytes 2018-12-18 10:38:06 INFO RandomKeyGenerator:266 - Starting progress bar Thread. 0.00% |█ | 0/1 Time: 0:00:002018-12-18 10:38:06 INFO RpcClient:250 - Creating Volume: vol-0-74492, with hadoop as owner and quota set to 1152921504606846976 bytes. 2018-12-18 10:38:06 INFO RpcClient:379 - Creating Bucket: vol-0-74492/bucket-0-16002, with Versioning false and Storage Type set to DISK 0.00% |█ | 0/1 Time: 0:02:402018-12-18 10:40:46 ERROR ChunkGroupOutputStream:275 - Try to allocate more blocks for write failed, already allocated 0 blocks for this write. 2018-12-18 10:40:46 ERROR RandomKeyGenerator:624 - Exception while adding key: key-0-28925 in bucket: org.apache.hadoop.ozone.client.OzoneBucket@1675c402 of volume: org.apache.hadoop.ozone.client.OzoneVolume@5b6bfafd. java.io.IOException: Allocate block failed, error:INTERNAL_ERROR at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.allocateBlock(OzoneManagerProtocolClientSideTranslatorPB.java:620) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.allocateNewBlock(ChunkGroupOutputStream.java:437) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:272) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleException(ChunkGroupOutputStream.java:377) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:473) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleFlushOrClose(ChunkGroupOutputStream.java:474) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.handleWrite(ChunkGroupOutputStream.java:309) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.write(ChunkGroupOutputStream.java:255) at org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49) at java.io.OutputStream.write(OutputStream.java:75) at org.apache.hadoop.ozone.freon.RandomKeyGenerator$OfflineProcessor.run(RandomKeyGenerator.java:606) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 0.00% |█ | 0/1 Time: 0:11:01 0.00% |█ | 0/1 Time: 0:11:19 {noformat} Here is the jstack for the freon process : [^jstack.txt] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-953) put key operation throws exception when run via ozone sh cli
Nilotpal Nandi created HDDS-953: --- Summary: put key operation throws exception when run via ozone sh cli Key: HDDS-953 URL: https://issues.apache.org/jira/browse/HDDS-953 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi steps taken : # created docker cluster consisting of 3 datanodes. # created volume and bucket and then ran put key operation. Put key operation throws following exception but the operation succeeds. {noformat} hadoop@4214b6a267fc:~$ ozone sh key put test-1/bbb-1/passwd /etc/passwd 2019-01-02 10:30:06 ERROR BlockOutputStream:530 - Unexpected Storage Container Exception: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: ContainerID 2 does not exist at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:495) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.validateResponse(BlockOutputStream.java:528) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.lambda$writeChunkToContainer$5(BlockOutputStream.java:605) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-01-02 10:30:06 ERROR BlockOutputStream:530 - Unexpected Storage Container Exception: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: ContainerID 2 does not exist at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:495) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.validateResponse(BlockOutputStream.java:528) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.lambda$handlePartialFlush$2(BlockOutputStream.java:386) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748){noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-954) get key operation is failing
Nilotpal Nandi created HDDS-954: --- Summary: get key operation is failing Key: HDDS-954 URL: https://issues.apache.org/jira/browse/HDDS-954 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Reporter: Nilotpal Nandi steps taken : --- # created 3 datanodes docker cluster. # created volume, bucket and keys. get key operations are failing. {noformat} hadoop@4214b6a267fc:~$ ozone fs -ls o3fs://bbb-1.test-1/ Found 21 items -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:30 o3fs://bbb-1.test-1/passwd -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:37 o3fs://bbb-1.test-1/passwd_1 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_10 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_11 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_12 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_13 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_14 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_15 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_16 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_17 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_18 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_19 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:37 o3fs://bbb-1.test-1/passwd_2 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_20 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:37 o3fs://bbb-1.test-1/passwd_3 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:37 o3fs://bbb-1.test-1/passwd_4 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:37 o3fs://bbb-1.test-1/passwd_5 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:37 o3fs://bbb-1.test-1/passwd_6 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:37 o3fs://bbb-1.test-1/passwd_7 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:37 o3fs://bbb-1.test-1/passwd_8 -rw-rw-rw- 1 hadoop hadoop 1010 2019-01-02 10:38 o3fs://bbb-1.test-1/passwd_9 hadoop@4214b6a267fc:~$ ozone fs -cat o3fs://bbb-1.test-1/passwd_9 cat: ContainerID 11 does not exist hadoop@4214b6a267fc:~$ ozone fs -cat o3fs://bbb-1.test-1/passwd_8 cat: ContainerID 10 does not exist hadoop@4214b6a267fc:~/logs$ ozone sh key get test-1/bbb-1/passwd_9 abc ContainerID 11 does not exist {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-988) containers remain in CLOSING state in one of the datanodes when there datanodes are isolated in docker cluster
Nilotpal Nandi created HDDS-988: --- Summary: containers remain in CLOSING state in one of the datanodes when there datanodes are isolated in docker cluster Key: HDDS-988 URL: https://issues.apache.org/jira/browse/HDDS-988 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode, SCM Reporter: Nilotpal Nandi Attachments: datanode_1, datanode_2, datanode_3, om, scm steps taken : --- # Created 3 datanodes docker cluster. # wrote some data to create a pipeline. # Then, isolated all datanodes , i.e, datanodes coud not communicate with each other . (datanodes can communicate with scm and om). # Tried to write some data again, write failed as expected. # After waiting for 'ozone.scm.stale.node.interval' and 'ozone.scm.dead.node.interval' , the container replicas are still in CLOSING state. Containers failed to get CLOSED. {noformat} hadoop@8876c7214ee5:~$ cat /data/hdds/hdds/40bb080a-1a9f-42c8-9e20-8257ed567e46/current/containerDir0/*/metadata/*.container ! checksum: 7ee8f706cf215a5fa4b7e9a195529c15147823ceea302ab4998c7476ee64ebf4 chunksPath: /data/hdds/hdds/40bb080a-1a9f-42c8-9e20-8257ed567e46/current/containerDir0/2/chunks containerDBType: RocksDB containerID: 2 containerType: KeyValueContainer layOutVersion: 1 maxSize: 5368709120 metadata: {} metadataPath: /data/hdds/hdds/40bb080a-1a9f-42c8-9e20-8257ed567e46/current/containerDir0/2/metadata originNodeId: 6e077f73-9fd9-4f4e-930f-578c9857912c originPipelineId: ee5f9e7a-0d63-412a-839a-77af2cf7ca93 state: CLOSING{noformat} Expectation : - The container should have at least two closed replicas . scm, om datanodes log attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-997) Add blockade Tests for scm isolation and mixed node isolation
Nilotpal Nandi created HDDS-997: --- Summary: Add blockade Tests for scm isolation and mixed node isolation Key: HDDS-997 URL: https://issues.apache.org/jira/browse/HDDS-997 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1027) Add blockade Tests for datanode isolation and scm failures
Nilotpal Nandi created HDDS-1027: Summary: Add blockade Tests for datanode isolation and scm failures Key: HDDS-1027 URL: https://issues.apache.org/jira/browse/HDDS-1027 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1036) container replica state in datanode should be QUASI-CLOSED if the datanode is isolated from other two datanodes in 3 datanode cluster
Nilotpal Nandi created HDDS-1036: Summary: container replica state in datanode should be QUASI-CLOSED if the datanode is isolated from other two datanodes in 3 datanode cluster Key: HDDS-1036 URL: https://issues.apache.org/jira/browse/HDDS-1036 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode, SCM Reporter: Nilotpal Nandi steps taken : --- # created a 3 datanode docker cluster. # wrote some data to create a pipeline. # Then, one of the datanodes is isolated from other two datanodes. All datanodes can communicate with SCM. # Tried to write new data , write failed. # Wait for 900 seconds. Observation: container state is CLOSED in all three replicas. Expectation: --- container state in isolated datanode should be QUASI-CLOSED. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1040) Add blockade Tests for client failures
Nilotpal Nandi created HDDS-1040: Summary: Add blockade Tests for client failures Key: HDDS-1040 URL: https://issues.apache.org/jira/browse/HDDS-1040 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1057) get key operation fails when client cannot communicate with 2 of the datanodes in 3 node cluster
Nilotpal Nandi created HDDS-1057: Summary: get key operation fails when client cannot communicate with 2 of the datanodes in 3 node cluster Key: HDDS-1057 URL: https://issues.apache.org/jira/browse/HDDS-1057 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi steps taken : -- # created 3 node docker cluster. # wrote a key # created partition such that 2 out of 3 datanodes cannot communicate with any other node. # Third datanode can communicate with all other nodes. # Tried to read the key Exception seen : {noformat} Failed to execute command cmdType: GetBlock E traceID: "9b3ebd93-e598-4ca2-a6f4-2389f2d35f63" E containerID: 22 E datanodeUuid: "15345663-15c9-4fe3-9b8f-a46123ba8a6e" E getBlock { E blockID { E containerID: 22 E localID: 101545011736215553 E blockCommitSequenceId: 5 E } E } E on datanode 15345663-15c9-4fe3-9b8f-a46123ba8a6e E java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception E at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) E at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) E at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:220) E at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:201) E at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118) E at org.apache.hadoop.ozone.client.io.KeyInputStream.getFromOmKeyInfo(KeyInputStream.java:305) E at org.apache.hadoop.ozone.client.rpc.RpcClient.getKey(RpcClient.java:608) E at org.apache.hadoop.ozone.client.OzoneBucket.readKey(OzoneBucket.java:284) E at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:95) E at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48) E at picocli.CommandLine.execute(CommandLine.java:919) E at picocli.CommandLine.access$700(CommandLine.java:104) E at picocli.CommandLine$RunLast.handle(CommandLine.java:1083) E at picocli.CommandLine$RunLast.handle(CommandLine.java:1051) E at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959) E at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242) E at picocli.CommandLine.parseWithHandler(CommandLine.java:1181) E at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61) E at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52) E at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:83) E Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception E at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526) E at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434) E at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) E at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) E at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) E at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678) E at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) E at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) E at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) E at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397) E at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) E at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) E at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) E at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) E at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584) E at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) E at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) E at java.util.concurrent.ThreadPoolExecutor.r
[jira] [Created] (HDDS-1067) freon run on client gets hung when two of the datanodes are down in 3 datanode cluster
Nilotpal Nandi created HDDS-1067: Summary: freon run on client gets hung when two of the datanodes are down in 3 datanode cluster Key: HDDS-1067 URL: https://issues.apache.org/jira/browse/HDDS-1067 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi steps taken : # created 3 node docker cluster. # wrote a key # created partition such that 2 out of 3 datanodes cannot communicate with any other node. # Third datanode can communicate with scm, om and the client. # ran freon to write key Observation : - freon run is hung. There is no timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1079) java.lang.RuntimeException: ManagedChannel allocation site exception seen on client cli when datanode restarted in one of the pipelines
Nilotpal Nandi created HDDS-1079: Summary: java.lang.RuntimeException: ManagedChannel allocation site exception seen on client cli when datanode restarted in one of the pipelines Key: HDDS-1079 URL: https://issues.apache.org/jira/browse/HDDS-1079 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi steps taken : # created 12 datanode cluster. # started put key operation with size 100GB. # Restarted one of the datanodes from one of the pipelines. exception seen on cli : {noformat} [root@ctr-e139-1542663976389-62237-01-06 ~]# time ozone sh key put volume1/bucket1/key1 /root/100G Feb 11, 2019 9:12:49 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=61, target=172.27.10.133:9858} was not shutdown properly!!! ~*~*~* Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. java.lang.RuntimeException: ManagedChannel allocation site at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) at org.apache.ratis.grpc.client.GrpcClientProtocolClient.(GrpcClientProtocolClient.java:116) at org.apache.ratis.grpc.client.GrpcClientRpc.lambda$new$0(GrpcClientRpc.java:54) at org.apache.ratis.util.PeerProxyMap$PeerAndProxy.lambda$getProxy$0(PeerProxyMap.java:60) at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:191) at org.apache.ratis.util.PeerProxyMap$PeerAndProxy.getProxy(PeerProxyMap.java:59) at org.apache.ratis.util.PeerProxyMap.getProxy(PeerProxyMap.java:106) at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequestAsync(GrpcClientRpc.java:69) at org.apache.ratis.client.impl.RaftClientImpl.sendRequestAsync(RaftClientImpl.java:324) at org.apache.ratis.client.impl.RaftClientImpl.sendRequestWithRetryAsync(RaftClientImpl.java:286) at org.apache.ratis.util.SlidingWindow$Client.sendOrDelayRequest(SlidingWindow.java:243) at org.apache.ratis.util.SlidingWindow$Client.retry(SlidingWindow.java:259) at org.apache.ratis.client.impl.RaftClientImpl.lambda$null$10(RaftClientImpl.java:293) at org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$0(TimeoutScheduler.java:85) at org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$1(TimeoutScheduler.java:104) at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:50) at org.apache.ratis.util.LogUtils$1.run(LogUtils.java:91) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Feb 11, 2019 9:12:49 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=29, target=172.27.10.133:9858} was not shutdown properly!!! ~*~*~* Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true. java.lang.RuntimeException: ManagedChannel allocation site at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:103) at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53) at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44) at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411) at org.apache.ratis.grpc.client.GrpcClientProtocolClient.(GrpcClientProtocolClient.java:116) at org.apache.ratis.grpc.client.GrpcClientRpc.lambda$new$0(GrpcClientRpc.java:54) at org.apache.ratis.util.PeerProxyMap$PeerAndProxy.lambda$getProxy$0(PeerProxyMap.java:60) at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:191) at org.apache.ratis.util.PeerProxyMap$PeerAndProxy.getProxy(PeerPr
[jira] [Created] (HDDS-1082) OutOfMemoryError while reading key
Nilotpal Nandi created HDDS-1082: Summary: OutOfMemoryError while reading key Key: HDDS-1082 URL: https://issues.apache.org/jira/browse/HDDS-1082 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken : # put key with size 100GB # Tried to read back the key. error thrown: -- {noformat} java.lang.OutOfMemoryError: Java heap space Dumping heap to /tmp/heapdump.bin ... Heap dump file created [3883178021 bytes in 10.667 secs] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at org.apache.ratis.thirdparty.com.google.protobuf.ByteString.toByteArray(ByteString.java:643) at org.apache.hadoop.ozone.common.Checksum.verifyChecksum(Checksum.java:217) at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:227) at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:188) at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:130) at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232) at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126) at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:49) at java.io.InputStream.read(InputStream.java:101) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100) at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98) at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48) at picocli.CommandLine.execute(CommandLine.java:919) at picocli.CommandLine.access$700(CommandLine.java:104) at picocli.CommandLine$RunLast.handle(CommandLine.java:1083) at picocli.CommandLine$RunLast.handle(CommandLine.java:1051) at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959) at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242) at picocli.CommandLine.parseWithHandler(CommandLine.java:1181) at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61) at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52) at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:83){noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1088) Add blockade Tests to test Replica Manager
Nilotpal Nandi created HDDS-1088: Summary: Add blockade Tests to test Replica Manager Key: HDDS-1088 URL: https://issues.apache.org/jira/browse/HDDS-1088 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi We need to add tests for testing Replica Manager for scenarios like loss of node, adding new nodes, under-replicated containers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1102) docker datanode stopped when new datanodes are added to the cluster
Nilotpal Nandi created HDDS-1102: Summary: docker datanode stopped when new datanodes are added to the cluster Key: HDDS-1102 URL: https://issues.apache.org/jira/browse/HDDS-1102 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken: # created 5 datanode cluster. # shutdown 2 datanodes # started the datanodes again. One of the datanodes was shut down. exception seen : {noformat} 2019-02-14 07:37:26 INFO LeaderElection:230 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8 got exception when requesting votes: {} java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found. at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214) at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146) at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102) Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found. at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233) at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214) at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139) at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265) at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:83) at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:187) at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-02-14 07:37:26 INFO LeaderElection:46 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8: Election PASSED; received 1 response(s) [6a0522ba-019e-4b77-ac1f-a9322cd525b8<-61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5#0:OK-t7] and 1 exception(s); 6a0522ba-019e-4b77-ac1f-a9322cd525b8:t7, leader=null, voted=6a0522ba-019e-4b77-ac1f-a9322cd525b8, raftlog=6a0522ba-019e-4b77-ac1f-a9322cd525b8-SegmentedRaftLog:OPENED, conf=3: [61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5:172.20.0.8:9858, 6a0522ba-019e-4b77-ac1f-a9322cd525b8:172.20.0.6:9858, 0f377918-aafa-4d8a-972a-6ead54048fba:172.20.0.3:9858], old=null 2019-02-14 07:37:26 INFO LeaderElection:52 - 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found. 2019-02-14 07:37:26 INFO RoleInfo:130 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8: shutdown LeaderElection 2019-02-14 07:37:26 INFO RaftServerImpl:161 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8 changes role from CANDIDATE to LEADER at term 7 for changeToLeader 2019-02-14 07:37:26 INFO RaftServerImpl:258 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8: change Leader from null to 6a0522ba-019e-4b77-ac1f-a9322cd525b8 at term 7 for becomeLeader, leader elected after 1066ms 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.staging.catchup.gap = 1000 (default) 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time = 25ms (default) 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout = 10s (default) 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout.denomination = 1s (default) 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.byte-limit = 33554432 (custom) 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.log.appender.buffer.element-limit = 1 (custom) 2019-02-14 07:37:26 INFO GrpcConfigKeys$Server:43 - raft.grpc.server.leader.outstanding.appends.max = 128 (default) 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.rpc.request.timeout = 3000ms (default) 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default) 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.log.appe
[jira] [Created] (HDDS-1124) java.lang.IllegalStateException exception in datanode log
Nilotpal Nandi created HDDS-1124: Summary: java.lang.IllegalStateException exception in datanode log Key: HDDS-1124 URL: https://issues.apache.org/jira/browse/HDDS-1124 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken : # created 12 datanodes cluster and running workload on all the nodes exception seen : --- {noformat} 2019-02-15 10:15:53,355 INFO org.apache.ratis.server.storage.RaftLogWorker: 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolled log segment from /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3036 to /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_3036-3047 2019-02-15 10:15:53,367 INFO org.apache.ratis.server.impl.RaftServerImpl: 943007c8-4fdd-4926-89e2-2c8c52c05073: set configuration 3048: [a40a7b01-a30b-469c-b373-9fcb20a126ed:172.27.54.212:9858, 8c77b16b-8054-49e3-b669-1ff759cfd271:172.27.23.196:9858, 943007c8-4fdd-4926-89e2-2c8c52c05073:172.27.76.72:9858], old=null at 3048 2019-02-15 10:15:53,523 INFO org.apache.ratis.server.storage.RaftLogWorker: 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: created new log segment /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3048 2019-02-15 10:15:53,580 ERROR org.apache.ratis.grpc.server.GrpcLogAppender: Failed onNext serverReply { requestorId: "943007c8-4fdd-4926-89e2-2c8c52c05073" replyId: "a40a7b01-a30b-469c-b373-9fcb20a126ed" raftGroupId { id: "\001\323\357*\221,O\300\200\266\001#C\327j\333" } success: true } term: 3 nextIndex: 3049 followerCommit: 3047 java.lang.IllegalStateException: reply's next index is 3049, request's previous is term: 1 index: 3047 at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60) at org.apache.ratis.grpc.server.GrpcLogAppender.onSuccess(GrpcLogAppender.java:285) at org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNextImpl(GrpcLogAppender.java:230) at org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:215) at org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:197) at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:421) at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33) at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33) at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519) at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-02-15 10:15:56,442 INFO org.apache.ratis.server.storage.RaftLogWorker: 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolling segment log-3048_3066 to index:3066 2019-02-15 10:15:56,442 INFO org.apache.ratis.server.storage.RaftLogWorker: 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolled log segment from /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3048 to /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_3048-3066 2019-02-15 10:15:56,564 INFO org.apache.ratis.server.storage.RaftLogWorker: 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: created new log segment /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3067 2019-02-15 10:16:45,420 INFO org.apache.ratis.server.storage.RaftLogWorker: 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolling segment log-3067_3077 to index:3077 {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1125) java.lang.InterruptedException seen in datanode logs
Nilotpal Nandi created HDDS-1125: Summary: java.lang.InterruptedException seen in datanode logs Key: HDDS-1125 URL: https://issues.apache.org/jira/browse/HDDS-1125 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken : # created 12 datanodes cluster and running workload on all the nodes exception seen : - {noformat} 2019-02-15 10:16:48,713 ERROR org.apache.ratis.server.impl.LogAppender: 943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3, i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232 java.lang.InterruptedException at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433) at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133) at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171) at org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152) at org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96) at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101) at java.lang.Thread.run(Thread.java:748) 2019-02-15 10:16:48,714 ERROR org.apache.ratis.server.impl.LogAppender: GrpcLogAppender(943007c8-4fdd-4926-89e2-2c8c52c05073 -> 8c77b16b-8054-49e3-b669-1ff759cfd271) hit IOException while loading raft log org.apache.ratis.server.storage.RaftLogIOException: 943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3, i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232 at org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:440) at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133) at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171) at org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152) at org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96) at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433) ... 6 more 2019-02-15 10:16:48,715 ERROR org.apache.ratis.server.impl.LogAppender: 943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3, i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232 java.lang.InterruptedException at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433) at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133) at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171) at org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152) at org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96) at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101) at java.lang.Thread.run(Thread.java:748) 2019-02-15 10:16:48,715 ERROR org.apache.ratis.server.impl.LogAppender: GrpcLogAppender(943007c8-4fdd-4926-89e2-2c8c52c05073 -> a40a7b01-a30b-469c-b373-9fcb20a126ed) hit IOException while loading raft log org.apache.ratis.server.storage.RaftLogIOException: 943007c8-4fdd-4926-89e2-2c8c52c05073: Failed readStateMachineData for (t:3, i:3084), STATEMACHINELOGENTRY, client-632E77ADA885, cid=6232 at org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:440) at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133) at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171) at org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152) at org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96) at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433) ... 6 more 2019-02-15 10:16:48,723 WARN org.apache.ratis.grpc.client.GrpcClientProtocolService: 943007c8-4fdd-4926-89e2-2c8c52c05073-5: onError: org.apache.ratis.thir
[jira] [Created] (HDDS-1126) datanode is trying to qausi-close a container which is already closed
Nilotpal Nandi created HDDS-1126: Summary: datanode is trying to qausi-close a container which is already closed Key: HDDS-1126 URL: https://issues.apache.org/jira/browse/HDDS-1126 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken : # created 12 datanodes cluster and running workload on all the nodes # running failure injection/restart on 1 datanode at a time periodically and randomly. Error seen in ozone.log : -- {noformat} 2019-02-18 06:06:32,780 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:176) - Executing cycle Number : 30 2019-02-18 06:06:32,784 [Command processor thread] DEBUG (CloseContainerCommandHandler.java:71) - Processing Close Container command. 2019-02-18 06:06:32,785 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:176) - Executing cycle Number : 31 2019-02-18 06:06:32,785 [Command processor thread] ERROR (CloseContainerCommandHandler.java:118) - Can't close container #37 org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Cannot quasi close container #37 while in CLOSED state. at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.quasiCloseContainer(KeyValueHandler.java:903) at org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.quasiCloseContainer(ContainerController.java:93) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:110) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:93) at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$1(DatanodeStateMachine.java:413) at java.lang.Thread.run(Thread.java:748) 2019-02-18 06:06:32,785 [Command processor thread] DEBUG (CloseContainerCommandHandler.java:71) - Processing Close Container command. 2019-02-18 06:06:32,788 [Command processor thread] DEBUG (CloseContainerCommandHandler.java:71) - Processing Close Container command. 2019-02-18 06:06:32,788 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:176) - Executing cycle Number : 32 2019-02-18 06:06:34,430 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:36,608 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:38,876 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:41,084 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:43,297 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:45,469 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:47,684 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:49,958 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:52,124 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:54,344 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:56,499 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:06:58,764 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:07:00,969 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:07:02,788 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:176) - Executing cycle Number : 33 2019-02-18 06:07:03,240 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2019-02-18 06:07:05,486 [main] DEBUG (OzoneClientFactory.java:287) - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1131) destroy pipeline failed with PipelineNotFoundException
Nilotpal Nandi created HDDS-1131: Summary: destroy pipeline failed with PipelineNotFoundException Key: HDDS-1131 URL: https://issues.apache.org/jira/browse/HDDS-1131 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken : # created 12 datanodes cluster and running workload on all the nodes exceptions seen in scm log {noformat} 2019-02-18 07:17:51,112 INFO org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils: destroying pipeline:PipelineID=01d3ef2a-912c-4fc0-80b6-012343d76adb with group-012343D76ADB:[a40a7b01-a30b-469c-b373-9fcb20a126ed:172.27.54.212:9858, 8c77b16b-8054-49e3-b669-1ff759cfd271:172.27.23.196:9858, 943007c8-4fdd-4926-89e2-2c8c52c05073:172.27.76.72:9858] 2019-02-18 07:17:51,112 INFO org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler: Close container Event triggered for container : #40 2019-02-18 07:17:51,113 INFO org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler: Close container Event triggered for container : #41 2019-02-18 07:17:51,114 INFO org.apache.hadoop.hdds.scm.container.CloseContainerEventHandler: Close container Event triggered for container : #42 2019-02-18 07:22:51,127 WARN org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=01d3ef2a-912c-4fc0-80b6-012343d76adb dn=a40a7b01-a30b-469c-b373-9fcb20a126ed{ip: 172.27.54.212, host: ctr-e139-1542663976389-62237-01-07.hwx.site} 2019-02-18 07:22:51,139 WARN org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=01d3ef2a-912c-4fc0-80b6-012343d76adb dn=8c77b16b-8054-49e3-b669-1ff759cfd271{ip: 172.27.23.196, host: ctr-e139-1542663976389-62237-01-15.hwx.site} 2019-02-18 07:22:51,149 WARN org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils: Pipeline destroy failed for pipeline=PipelineID=01d3ef2a-912c-4fc0-80b6-012343d76adb dn=943007c8-4fdd-4926-89e2-2c8c52c05073{ip: 172.27.76.72, host: ctr-e139-1542663976389-62237-01-06.hwx.site} 2019-02-18 07:22:51,150 ERROR org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils: Destroy pipeline failed for pipeline:PipelineID=01d3ef2a-912c-4fc0-80b6-012343d76adb with group-012343D76ADB:[a40a7b01-a30b-469c-b373-9fcb20a126ed:172.27.54.212:9858, 8c77b16b-8054-49e3-b669-1ff759cfd271:172.27.23.196:9858, 943007c8-4fdd-4926-89e2-2c8c52c05073:172.27.76.72:9858] org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=01d3ef2a-912c-4fc0-80b6-012343d76adb not found at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:112) at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.removePipeline(PipelineStateMap.java:247) at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManager.removePipeline(PipelineStateManager.java:90) at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removePipeline(SCMPipelineManager.java:261) at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils.destroyPipeline(RatisPipelineUtils.java:103) at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils.lambda$finalizeAndDestroyPipeline$1(RatisPipelineUtils.java:133) at org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$0(TimeoutScheduler.java:85) at org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$1(TimeoutScheduler.java:104) at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:50) at org.apache.ratis.util.LogUtils$1.run(LogUtils.java:91) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748){noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1164) Add New blockade Tests to test Replica Manager
Nilotpal Nandi created HDDS-1164: Summary: Add New blockade Tests to test Replica Manager Key: HDDS-1164 URL: https://issues.apache.org/jira/browse/HDDS-1164 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1206) need to handle in the client when one of the datanode disk goes out of space
Nilotpal Nandi created HDDS-1206: Summary: need to handle in the client when one of the datanode disk goes out of space Key: HDDS-1206 URL: https://issues.apache.org/jira/browse/HDDS-1206 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi Assignee: Shashikant Banerjee steps taken : # create 40 datanode cluster. # one of the datanodes has less than 5 GB space. # Started writing key of size 600MB. operation failed: Error on the client: {noformat} Fri Mar 1 09:05:28 UTC 2019 Ruuning /root/hadoop_trunk/ozone-0.4.0-SNAPSHOT/bin/ozone sh key put testvol172275910-1551431122-1/testbuck172275910-1551431122-1/test_file24 /root/test_files/test_file24 original md5sum a6de00c9284708585f5a99b0490b0b23 2019-03-01 09:05:39,142 ERROR storage.BlockOutputStream: Unexpected Storage Container Exception: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: ContainerID 79 creation failed at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:568) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.validateResponse(BlockOutputStream.java:535) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.lambda$writeChunkToContainer$5(BlockOutputStream.java:613) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-03-01 09:05:39,578 ERROR storage.BlockOutputStream: Unexpected Storage Container Exception: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: ContainerID 79 creation failed at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:568) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.validateResponse(BlockOutputStream.java:535) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.lambda$writeChunkToContainer$5(BlockOutputStream.java:613) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-03-01 09:05:40,368 ERROR storage.BlockOutputStream: Unexpected Storage Container Exception: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: ContainerID 79 creation failed at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:568) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.validateResponse(BlockOutputStream.java:535) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.lambda$writeChunkToContainer$5(BlockOutputStream.java:613) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-03-01 09:05:40,450 ERROR storage.BlockOutputStream: Unexpected Storage Container Exception: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: ContainerID 79 creation failed at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:568) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.validateResponse(BlockOutputStream.java:535) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.lambda$writeChunkToContainer$5(BlockOutputStream.java:613) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20
[jira] [Created] (HDDS-1251) all chunks are not deleted by block deletion even when all keys are deleted and all containers are closed
Nilotpal Nandi created HDDS-1251: Summary: all chunks are not deleted by block deletion even when all keys are deleted and all containers are closed Key: HDDS-1251 URL: https://issues.apache.org/jira/browse/HDDS-1251 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken : --- # created 40 nodes cluster, wrote data on all datanodes. # deleted all keys from the cluster and all containers are closed. block deletion triggered and deleted most of the chunks from all datanodes. But , it could not delete all chunks even after several days. expectations : all chunks should be deleted if there is no key present in the cluster and all containers are closed -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1289) get Key failed on SCM restart
Nilotpal Nandi created HDDS-1289: Summary: get Key failed on SCM restart Key: HDDS-1289 URL: https://issues.apache.org/jira/browse/HDDS-1289 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi Attachments: hadoop-hdfs-scm-ctr-e139-1542663976389-86524-01-03.log Seeing ContainerNotFoundException in scm log when get key operation tried after scm restart. scm.log: [^hadoop-hdfs-scm-ctr-e139-1542663976389-86524-01-03.log] {noformat} 2019-03-13 17:00:54,348 ERROR container.ContainerReportHandler (ContainerReportHandler.java:processContainerReplicas(173)) - Received container report for an unknown container 22 from datanode 80f046cb-6fe2-4a05-bb67-9bf46f48723b{ip: 172.27.69.155, host: ctr-e139-1542663976389-86524-01-05.hwx.site} {} org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #22 at org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:543) at org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.updateContainerReplica(ContainerStateMap.java:230) at org.apache.hadoop.hdds.scm.container.ContainerStateManager.updateContainerReplica(ContainerStateManager.java:565) at org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerReplica(SCMContainerManager.java:393) at org.apache.hadoop.hdds.scm.container.ReportHandlerHelper.processContainerReplica(ReportHandlerHelper.java:74) at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:159) at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:110) at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:51) at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-03-13 17:00:54,349 ERROR container.ContainerReportHandler (ContainerReportHandler.java:processContainerReplicas(173)) - Received container report for an unknown container 23 from datanode 80f046cb-6fe2-4a05-bb67-9bf46f48723b{ip: 172.27.69.155, host: ctr-e139-1542663976389-86524-01-05.hwx.site} {} org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #23 at org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:543) at org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.updateContainerReplica(ContainerStateMap.java:230) at org.apache.hadoop.hdds.scm.container.ContainerStateManager.updateContainerReplica(ContainerStateManager.java:565) at org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerReplica(SCMContainerManager.java:393) at org.apache.hadoop.hdds.scm.container.ReportHandlerHelper.processContainerReplica(ReportHandlerHelper.java:74) at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:159) at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:110) at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:51) at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-03-13 17:01:24,230 ERROR container.ContainerReportHandler (ContainerReportHandler.java:processContainerReplicas(173)) - Received container report for an unknown container 22 from datanode 076fd0d8-ab5f-4fbe-ad10-b71a1ccb19bf{ip: 172.27.39.88, host: ctr-e139-1542663976389-86524-01-04.hwx.site} {} org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #22 at org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:543) at org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.updateContainerReplica(ContainerStateMap.java:230) at org.apache.hadoop.hdds.scm.container.ContainerStateManager.updateContainerReplica(ContainerStateManager.java:565) at org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerReplica(SCMContainerManager.java:393) at org.apache.hadoop.hdds.scm.container.ReportHandlerHelper.processContainerReplica(ReportHandlerHelper.java:74) at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:159) at org.apache.hadoop.hdds.scm.container.ContainerR
[jira] [Created] (HDDS-1290) ozone.log is not getting created in logs directory
Nilotpal Nandi created HDDS-1290: Summary: ozone.log is not getting created in logs directory Key: HDDS-1290 URL: https://issues.apache.org/jira/browse/HDDS-1290 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi ozone.log is getting created in the log directory of the client or any other nodes of ozone cluster. ozone version : Source code repository g...@github.com:hortonworks/ozone.git -r 67b7c4fd071b3f557bdb54be2a266b8a611cbce6 Compiled by jenkins on 2019-03-06T22:02Z Compiled with protoc 2.5.0 >From source with checksum 65be9a337d178cd3855f5c5a2f111 Using HDDS 0.4.0.3.0.100.0-348 Source code repository g...@github.com:hortonworks/ozone.git -r 67b7c4fd071b3f557bdb54be2a266b8a611cbce6 Compiled by jenkins on 2019-03-06T22:01Z Compiled with protoc 2.5.0 >From source with checksum 324109cb3e8b188c1b89dc0b328c3a [root@ctr-e139-1542663976389-86524-01-06 hdfs]# hadoop version Hadoop 3.1.1.3.0.100.0-348 Source code repository g...@github.com:hortonworks/hadoop.git -r 484434b1c2480bdc9314a7ee1ade8a0f4db1758f Compiled by jenkins on 2019-03-06T22:14Z Compiled with protoc 2.5.0 >From source with checksum ba6aad94c14256ef3ad8634e3b5086 This command was run using /usr/hdp/3.0.100.0-348/hadoop/hadoop-common-3.1.1.3.0.100.0-348.jar -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1298) blockade tests failing as the nodes are not able to communicate with Ozone Manager
Nilotpal Nandi created HDDS-1298: Summary: blockade tests failing as the nodes are not able to communicate with Ozone Manager Key: HDDS-1298 URL: https://issues.apache.org/jira/browse/HDDS-1298 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken: # started 3 datanodes docker cluster. # freon run fails with error : "No such service: ozoneManager" {noformat} om_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-15T17:34Z om_1 | STARTUP_MSG: java = 11.0.1 om_1 | / om_1 | 2019-03-18 06:31:41 INFO OzoneManager:51 - registered UNIX signal handlers for [TERM, HUP, INT] om_1 | 2019-03-18 06:31:41 WARN ScmUtils:77 - ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. om_1 | 2019-03-18 06:31:41 INFO OzoneManager:484 - OM Service ID is not set. Setting it to the default ID: omServiceIdDefault om_1 | 2019-03-18 06:31:41 INFO OzoneManager:490 - OM Node ID is not set. Setting it to the OmStorage's OmID: 25501758-f7f6-42d5-8196-52a885af7e23 om_1 | 2019-03-18 06:31:41 INFO OzoneManager:441 - Found matching OM address with OMServiceId: null, OMNodeId: null, RPC Address: om:9862 and Ratis port: 9872 om_1 | 2019-03-18 06:31:42 WARN ScmUtils:77 - ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. om_1 | 2019-03-18 06:31:42 INFO log:192 - Logging initialized @4061ms om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: userTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column profile:DBProfile.DISK for Table:userTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: volumeTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column profile:DBProfile.DISK for Table:volumeTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: bucketTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column profile:DBProfile.DISK for Table:bucketTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: keyTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column profile:DBProfile.DISK for Table:keyTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: deletedTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column profile:DBProfile.DISK for Table:deletedTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: openKeyTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column profile:DBProfile.DISK for Table:openKeyTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: s3Table om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column profile:DBProfile.DISK for Table:s3Table om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: multipartInfoTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column profile:DBProfile.DISK for Table:multipartInfoTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: s3SecretTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column profile:DBProfile.DISK for Table:s3SecretTable om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for table: default om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:158 - Using default column profile:DBProfile.DISK for Table:default om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:189 - Using default options. DBProfile.DISK om_1 | 2019-03-18 06:31:42 INFO CallQueueManager:84 - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 2000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. om_1 | 2019-03-18 06:31:42 INFO Server:1074 - Starting Socket Reader #1 for port 9862 om_1 | 2019-03-18 06:31:43 WARN ScmUtils:77 - ozone.om.db.dirs is not configured. We recommend adding this setting. Falling back to ozone.metadata.dirs instead. om_1 | 2019-03-18 06:31:43 INFO OzoneManager:1129 - OzoneManager RPC server is listening at om/172.21.0.3:9862 om_1 | 2019-03-18 06:31:43 INFO MetricsConfig:118 - Loaded properties from hadoop-metrics2.properties om_1 | 2019-03-18 06:31:43 INFO MetricsSystemImpl:374 - Scheduled Metric snapshot period at 10 second(s). om_1 | 2019-03-18 06:31:43 INFO MetricsSystemImpl:191 - OzoneManager metrics system started om_1 | 2019-03-18 06:31:43 INFO Server:1314 - IPC Server Responder: starting om_1 | 2019-03-18 06:31:43
[jira] [Created] (HDDS-1325) Exception thrown while initializing ozoneClientAdapter
Nilotpal Nandi created HDDS-1325: Summary: Exception thrown while initializing ozoneClientAdapter Key: HDDS-1325 URL: https://issues.apache.org/jira/browse/HDDS-1325 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi ozone version : {noformat} Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r 568d3ab8b65d1348dec9c971feffe200e6cba2ef Compiled by nnandi on 2019-03-19T03:54Z Compiled with protoc 2.5.0 >From source with checksum c44d339e20094d3054754078afbf4c Using HDDS 0.5.0-SNAPSHOT Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r 568d3ab8b65d1348dec9c971feffe200e6cba2ef Compiled by nnandi on 2019-03-19T03:53Z Compiled with protoc 2.5.0 >From source with checksum b354934fb1352f4d5425114bf8dce11 {noformat} steps taken : --- # Add ozone libs in hadoop classpath. # Tried to run s3dupdo workload ([https://github.com/t3rmin4t0r/s3dupdo]) Here is the exception thrown : {noformat} java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.fs.ozone.OzoneClientAdapterFactory.lambda$createAdapter$1(OzoneClientAdapterFactory.java:65) at org.apache.hadoop.fs.ozone.OzoneClientAdapterFactory.createAdapter(OzoneClientAdapterFactory.java:105) at org.apache.hadoop.fs.ozone.OzoneClientAdapterFactory.createAdapter(OzoneClientAdapterFactory.java:61) at org.apache.hadoop.fs.ozone.OzoneFileSystem.initialize(OzoneFileSystem.java:167) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3326) at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:532) at org.notmysock.repl.Works$CopyWorker.run(Works.java:243) at org.notmysock.repl.Works$CopyWorker.call(Works.java:279) at org.notmysock.repl.Works$CopyWorker.call(Works.java:204) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.LinkageError: loader constraint violation: loader (instance of org/apache/hadoop/fs/ozone/FilteredClassLoader) previously initiated loading for a different type with name "org/apache/hadoop/security/token/Token" at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.fs.ozone.FilteredClassLoader.loadClass(FilteredClassLoader.java:71) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetPublicMethods(Class.java:2902) at java.lang.Class.getMethods(Class.java:1615) at sun.misc.ProxyGenerator.generateClassFile(ProxyGenerator.java:451) at sun.misc.ProxyGenerator.generateProxyClass(ProxyGenerator.java:339) at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:639) at java.lang.reflect.Proxy$ProxyClassFactory.apply(Proxy.java:557) at java.lang.reflect.WeakCache$Factory.get(WeakCache.java:230) at java.lang.reflect.WeakCache.get(WeakCache.java:127) at java.lang.reflect.Proxy.getProxyClass0(Proxy.java:419) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:719) at org.apache.hadoop.ozone.client.OzoneClientFactory.getClient(OzoneClientFactory.java:264) at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169) at org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:140) at org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:104) at org.apache.hadoop.fs.ozone.OzoneClientAdapterImpl.(OzoneClientAdapterImpl.java:75) ... 20 more{noformat}
[jira] [Created] (HDDS-1326) putkey operation failed with java.lang.ArrayIndexOutOfBoundsException
Nilotpal Nandi created HDDS-1326: Summary: putkey operation failed with java.lang.ArrayIndexOutOfBoundsException Key: HDDS-1326 URL: https://issues.apache.org/jira/browse/HDDS-1326 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken : --- # trying to write key in 40 node cluster. # write failed. client output --- {noformat} e530-491c-ab03-3b1c34d1a751:c80390, 974a806d-bf7d-4f1b-adb4-d51d802d368a:c80390, 469bd8c4-5da2-43bb-bc4b-7edd884931e5:c80390] 2019-03-22 10:56:19,592 [main] WARN - Encountered exception {} java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.ExecutionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.StateMachineException: org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException from Server 5d3eb91f-e530-491c-ab03-3b1c34d1a751: Container 1269 in CLOSED state at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.close(BlockOutputStream.java:511) at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.close(BlockOutputStreamEntry.java:144) at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:565) at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:329) at org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:273) at org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:96) at org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:111) at org.apache.hadoop.ozone.web.ozShell.keys.PutKeyHandler.call(PutKeyHandler.java:53) at picocli.CommandLine.execute(CommandLine.java:919) at picocli.CommandLine.access$700(CommandLine.java:104) at picocli.CommandLine$RunLast.handle(CommandLine.java:1083) at picocli.CommandLine$RunLast.handle(CommandLine.java:1051) at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959) at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242) at picocli.CommandLine.parseWithHandler(CommandLine.java:1181) at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61) at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:82) at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52) at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:93) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.StateMachineException: org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException from Server 5d3eb91f-e530-491c-ab03-3b1c34d1a751: Container 1269 in CLOSED state at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.waitOnFlushFutures(BlockOutputStream.java:529) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.handleFlush(BlockOutputStream.java:481) at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.close(BlockOutputStream.java:496) ... 19 more Caused by: java.util.concurrent.CompletionException: org.apache.ratis.protocol.StateMachineException: org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException from Server 5d3eb91f-e530-491c-ab03-3b1c34d1a751: Container 1269 in CLOSED state at org.apache.ratis.client.impl.RaftClientImpl.handleStateMachineException(RaftClientImpl.java:402) at org.apache.ratis.client.impl.RaftClientImpl.lambda$sendAsync$3(RaftClientImpl.java:198) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.ratis.client.impl.RaftClientImpl$PendingAsyncRequest.setReply(RaftClientImpl.java:95) at org.apache.ratis.client.impl.RaftClientImpl$PendingAsyncRequest.setReply(RaftClientImpl.java:75) at org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:127) at org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:279) at org.apache.ratis.client.impl.RaftClientImpl.lambda$sendRequestAsync$13(RaftClientImpl.java:344) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.rati
[jira] [Created] (HDDS-1338) ozone shell commands are throwing InvocationTargetException
Nilotpal Nandi created HDDS-1338: Summary: ozone shell commands are throwing InvocationTargetException Key: HDDS-1338 URL: https://issues.apache.org/jira/browse/HDDS-1338 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi ozone version {noformat} Source code repository g...@github.com:hortonworks/ozone.git -r 310ebf5dc83b6c9e68d09246ed6c6f7cf6370fde Compiled by jenkins on 2019-03-21T22:06Z Compiled with protoc 2.5.0 >From source with checksum 9c367143ad43b81ca84bfdaafd1c3f Using HDDS 0.4.0.3.0.100.0-388 Source code repository g...@github.com:hortonworks/ozone.git -r 310ebf5dc83b6c9e68d09246ed6c6f7cf6370fde Compiled by jenkins on 2019-03-21T22:06Z Compiled with protoc 2.5.0 >From source with checksum f3297cbd3a5f59fb4e5fd551afa05ba9 {noformat} Here is the ozone volume create failure output : {noformat} hdfs@ctr-e139-1542663976389-91321-01-02 ~]$ ozone sh volume create testvolume11 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-388/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-388/hadoop-ozone/share/ozone/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 19/03/26 17:31:37 ERROR client.OzoneClientFactory: Couldn't create protocol class org.apache.hadoop.ozone.client.rpc.RpcClient exception: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:291) at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:169) at org.apache.hadoop.ozone.web.ozShell.OzoneAddress.createClient(OzoneAddress.java:111) at org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:70) at org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.call(CreateVolumeHandler.java:38) at picocli.CommandLine.execute(CommandLine.java:919) at picocli.CommandLine.access$700(CommandLine.java:104) at picocli.CommandLine$RunLast.handle(CommandLine.java:1083) at picocli.CommandLine$RunLast.handle(CommandLine.java:1051) at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959) at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242) at picocli.CommandLine.parseWithHandler(CommandLine.java:1181) at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61) at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:82) at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52) at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:93) Caused by: java.lang.VerifyError: Cannot inherit from final class at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.(OzoneManagerProtocolClientSideTranslatorPB.java:169) at org.apache.hadoop.ozone.client.rpc.RpcClient.(RpcClient.java:142) ... 20 more Couldn't create protocol class org.apache.hadoop.ozone.client.rpc.RpcClient {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1298) blockade tests failing as the nodes are not able to communicate with Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nilotpal Nandi resolved HDDS-1298. -- Resolution: Duplicate > blockade tests failing as the nodes are not able to communicate with Ozone > Manager > -- > > Key: HDDS-1298 > URL: https://issues.apache.org/jira/browse/HDDS-1298 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Nilotpal Nandi >Assignee: Nilotpal Nandi >Priority: Critical > Attachments: alllogs.log > > > steps taken: > > # started 3 datanodes docker cluster. > # freon run fails with error : "No such service: ozoneManager" > > {noformat} > om_1 | STARTUP_MSG: build = https://github.com/apache/hadoop.git -r > e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on > 2019-01-15T17:34Z > om_1 | STARTUP_MSG: java = 11.0.1 > om_1 | / > om_1 | 2019-03-18 06:31:41 INFO OzoneManager:51 - registered UNIX signal > handlers for [TERM, HUP, INT] > om_1 | 2019-03-18 06:31:41 WARN ScmUtils:77 - ozone.om.db.dirs is not > configured. We recommend adding this setting. Falling back to > ozone.metadata.dirs instead. > om_1 | 2019-03-18 06:31:41 INFO OzoneManager:484 - OM Service ID is not set. > Setting it to the default ID: omServiceIdDefault > om_1 | 2019-03-18 06:31:41 INFO OzoneManager:490 - OM Node ID is not set. > Setting it to the OmStorage's OmID: 25501758-f7f6-42d5-8196-52a885af7e23 > om_1 | 2019-03-18 06:31:41 INFO OzoneManager:441 - Found matching OM address > with OMServiceId: null, OMNodeId: null, RPC Address: om:9862 and Ratis port: > 9872 > om_1 | 2019-03-18 06:31:42 WARN ScmUtils:77 - ozone.om.db.dirs is not > configured. We recommend adding this setting. Falling back to > ozone.metadata.dirs instead. > om_1 | 2019-03-18 06:31:42 INFO log:192 - Logging initialized @4061ms > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: userTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column > profile:DBProfile.DISK for Table:userTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: volumeTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column > profile:DBProfile.DISK for Table:volumeTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: bucketTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column > profile:DBProfile.DISK for Table:bucketTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: keyTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column > profile:DBProfile.DISK for Table:keyTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: deletedTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column > profile:DBProfile.DISK for Table:deletedTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: openKeyTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column > profile:DBProfile.DISK for Table:openKeyTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: s3Table > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column > profile:DBProfile.DISK for Table:s3Table > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: multipartInfoTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column > profile:DBProfile.DISK for Table:multipartInfoTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: s3SecretTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:152 - Using default column > profile:DBProfile.DISK for Table:s3SecretTable > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:101 - using custom profile for > table: default > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:158 - Using default column > profile:DBProfile.DISK for Table:default > om_1 | 2019-03-18 06:31:42 INFO DBStoreBuilder:189 - Using default options. > DBProfile.DISK > om_1 | 2019-03-18 06:31:42 INFO CallQueueManager:84 - Using callQueue: class > java.util.concurrent.LinkedBlockingQueue, queueCapacity: 2000, scheduler: > class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. > om_1 | 2019-03-18 06:31:42 INFO Server:1074 - Starting Socket Reader #1 for > port 9862 > om_1 | 2019-03-18 06:31:43 WARN ScmUtils:77 - ozone.om.db.dirs is not > configured. We recommend adding this setting. Falling back to > ozone.metadata.dirs instead. > om_1 | 2019-03-18 06:31:43 INF
[jira] [Created] (HDDS-1497) Refactor blockade Tests
Nilotpal Nandi created HDDS-1497: Summary: Refactor blockade Tests Key: HDDS-1497 URL: https://issues.apache.org/jira/browse/HDDS-1497 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi Assignee: Nilotpal Nandi -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1534) freon should return non-zero exit code on failure
Nilotpal Nandi created HDDS-1534: Summary: freon should return non-zero exit code on failure Key: HDDS-1534 URL: https://issues.apache.org/jira/browse/HDDS-1534 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi Assignee: Nilotpal Nandi Currently freon does not return any non-zero exit code even on failure. The status shows as "Failed" but the exit code is always zero. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2043) "VOLUME_NOT_FOUND" exception thrown while listing volumes
Nilotpal Nandi created HDDS-2043: Summary: "VOLUME_NOT_FOUND" exception thrown while listing volumes Key: HDDS-2043 URL: https://issues.apache.org/jira/browse/HDDS-2043 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone CLI, Ozone Manager Reporter: Nilotpal Nandi ozone list volume command throws OMException /opt/cloudera/parcels/CDH/bin/ozone sh volume list --user root VOLUME_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Volume info not found for vol-test-putfile-1566902803 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2350) NullPointerException seen in datanode log while writing data
Nilotpal Nandi created HDDS-2350: Summary: NullPointerException seen in datanode log while writing data Key: HDDS-2350 URL: https://issues.apache.org/jira/browse/HDDS-2350 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi NullPointerException exception seen in datanode log while writing 10GB data. There is one pipelinee with factor 3 while writing data. {noformat} 2019-10-23 11:25:45,674 ERROR org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: Error getting metrics from source ratis_core.ratis_leader.a23fb300-4c1e-420f-a21e-7e73d0c22cbe@group-4CA404C938C2 java.lang.NullPointerException at org.apache.ratis.server.impl.RaftLeaderMetrics.lambda$null$2(RaftLeaderMetrics.java:86) at com.github.joshelser.dropwizard.metrics.hadoop.HadoopMetrics2Reporter.snapshotAllMetrics(HadoopMetrics2Reporter.java:239) at com.github.joshelser.dropwizard.metrics.hadoop.HadoopMetrics2Reporter.getMetrics(HadoopMetrics2Reporter.java:219) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.snapshotMetrics(MetricsSystemImpl.java:419) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.sampleMetrics(MetricsSystemImpl.java:406) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.onTimerEvent(MetricsSystemImpl.java:381) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$4.run(MetricsSystemImpl.java:368) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) 2019-10-23 11:25:55,673 ERROR org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: Error getting metrics from source ratis_core.ratis_leader.a23fb300-4c1e-420f-a21e-7e73d0c22cbe@group-4CA404C938C2 java.lang.NullPointerException at org.apache.ratis.server.impl.RaftLeaderMetrics.lambda$null$2(RaftLeaderMetrics.java:86) at com.github.joshelser.dropwizard.metrics.hadoop.HadoopMetrics2Reporter.snapshotAllMetrics(HadoopMetrics2Reporter.java:239) at com.github.joshelser.dropwizard.metrics.hadoop.HadoopMetrics2Reporter.getMetrics(HadoopMetrics2Reporter.java:219) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.snapshotMetrics(MetricsSystemImpl.java:419) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.sampleMetrics(MetricsSystemImpl.java:406) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.onTimerEvent(MetricsSystemImpl.java:381) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$4.run(MetricsSystemImpl.java:368) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) 2019-10-23 11:26:05,674 ERROR org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: Error getting metrics from source ratis_core.ratis_leader.a23fb300-4c1e-420f-a21e-7e73d0c22cbe@group-4CA404C938C2 java.lang.NullPointerException at org.apache.ratis.server.impl.RaftLeaderMetrics.lambda$null$2(RaftLeaderMetrics.java:86) at com.github.joshelser.dropwizard.metrics.hadoop.HadoopMetrics2Reporter.snapshotAllMetrics(HadoopMetrics2Reporter.java:239) at com.github.joshelser.dropwizard.metrics.hadoop.HadoopMetrics2Reporter.getMetrics(HadoopMetrics2Reporter.java:219) at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.snapshotMetrics(MetricsSystemImpl.java:419) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.sampleMetrics(MetricsSystemImpl.java:406) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.onTimerEvent(MetricsSystemImpl.java:381) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl$4.run(MetricsSystemImpl.java:368) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505){noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14980) diskbalancer query command always tries to contact to port 9867
Nilotpal Nandi created HDFS-14980: - Summary: diskbalancer query command always tries to contact to port 9867 Key: HDFS-14980 URL: https://issues.apache.org/jira/browse/HDFS-14980 Project: Hadoop HDFS Issue Type: Bug Components: diskbalancer Reporter: Nilotpal Nandi disbalancer query commands always tries to connect to port 9867 even when datanode IPC port is different. In this setup , datanode IPC port is set to 20001. diskbalancer report command works fine and connects to IPC port 20001 {noformat} hdfs diskbalancer -report -node 172.27.131.193 19/11/12 08:58:55 INFO command.Command: Processing report command 19/11/12 08:58:57 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 19/11/12 08:58:57 INFO block.BlockTokenSecretManager: Setting block keys 19/11/12 08:58:57 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec 19/11/12 08:58:58 INFO command.Command: Reporting volume information for DataNode(s). These DataNode(s) are parsed from '172.27.131.193'. Processing report command Reporting volume information for DataNode(s). These DataNode(s) are parsed from '172.27.131.193'. [172.27.131.193:20001] - : 3 volumes with node data density 0.05. [DISK: volume-/dataroot/ycloud/dfs/NEW_DISK1/] - 0.15 used: 39343871181/259692498944, 0.85 free: 220348627763/259692498944, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False. [DISK: volume-/dataroot/ycloud/dfs/NEW_DISK2/] - 0.15 used: 39371179986/259692498944, 0.85 free: 220321318958/259692498944, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False. [DISK: volume-/dataroot/ycloud/dfs/dn/] - 0.19 used: 49934903670/259692498944, 0.81 free: 209757595274/259692498944, isFailed: False, isReadOnly: False, isSkip: False, isTransient: False. {noformat} But diskbalancer query command fails and tries to connect to port 9867 (default port). {noformat} hdfs diskbalancer -query 172.27.131.193 19/11/12 06:37:15 INFO command.Command: Executing "query plan" command. 19/11/12 06:37:16 INFO ipc.Client: Retrying connect to server: /172.27.131.193:9867. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 19/11/12 06:37:17 INFO ipc.Client: Retrying connect to server: /172.27.131.193:9867. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) .. .. .. 19/11/12 06:37:25 ERROR tools.DiskBalancerCLI: Exception thrown while running DiskBalancerCLI. {noformat} Expectation : diskbalancer query command should work fine without explicitly mentioning datanode IPC port address -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-2604) scmcli pipeline deactivate command not working
Nilotpal Nandi created HDDS-2604: Summary: scmcli pipeline deactivate command not working Key: HDDS-2604 URL: https://issues.apache.org/jira/browse/HDDS-2604 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Client Reporter: Nilotpal Nandi Assignee: Nilotpal Nandi scmcli pipeline deactivate not working output : {noformat} ozone scmcli pipeline deactivate 212e1f47-4890-49c2-a950-4d0b3a70cbfd Unknown command type: DeactivatePipeline root@st-ozone-kg2qce-l2ltm:/ansible# echo $? 255{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15619) Metric for ordered snapshot deletion GC thread
Nilotpal Nandi created HDFS-15619: - Summary: Metric for ordered snapshot deletion GC thread Key: HDFS-15619 URL: https://issues.apache.org/jira/browse/HDFS-15619 Project: Hadoop HDFS Issue Type: Task Components: hdfs Reporter: Nilotpal Nandi Assignee: Nilotpal Nandi Following info should be captured and shown in JMX for garbage collection thread of ordered snapshot deletion * metric for all pending snapshots to be GCed * Number of times GC thread ran * Number of Snapshots already GCed * Average time taken by each GC run * Thread running Status * metric for failed deletion of GC thread -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-206) default port number taken by ksm is 9862 while listing the volumes
Nilotpal Nandi created HDDS-206: --- Summary: default port number taken by ksm is 9862 while listing the volumes Key: HDDS-206 URL: https://issues.apache.org/jira/browse/HDDS-206 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Reporter: Nilotpal Nandi Fix For: 0.2.1 Here is the output of ozone -listVolume command without any port mentioned . By default, it chooses the port number as 9862 which is not mentioned in the ozone-site.xml {noformat} [root@ozone-vm bin]# ./ozone oz -listVolume o3://127.0.0.1/ 2018-06-29 04:42:20,652 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-06-29 04:42:21,914 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:22,915 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:23,917 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:24,925 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:25,928 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:26,931 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:27,932 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:28,934 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:29,935 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:30,938 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:9862. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2018-06-29 04:42:31,075 [main] ERROR - Couldn't create protocol class org.apache.hadoop.ozone.client.rpc.RpcClient exception: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:292) at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:172) at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:156) at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:111) at org.apache.hadoop.ozone.web.ozShell.Handler.verifyURI(Handler.java:96) at org.apache.hadoop.ozone.web.ozShell.volume.ListVolumeHandler.execute(ListVolumeHandler.java:80) at org.apache.hadoop.ozone.web.ozShell.Shell.dispatch(Shell.java:395) at org.apache.hadoop.ozone.web.ozShell.Shell.run(Shell.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:114) Caused by: java.net.ConnectException: Call From ozone-vm/10.200.5.166 to localhost:9862 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Co
[jira] [Created] (HDDS-207) ozone listVolume command accepts random values as argument
Nilotpal Nandi created HDDS-207: --- Summary: ozone listVolume command accepts random values as argument Key: HDDS-207 URL: https://issues.apache.org/jira/browse/HDDS-207 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi Fix For: 0.2.1 When argument from listVolume is provided, it complains. But a random argument is provided for listVolume command, it accepts and displays all the volumes. {noformat} [root@ozone-vm bin]# ./ozone oz -listVolume Missing argument for option: listVolumeERROR: null [root@ozone-vm bin]# ./ozone oz -listVolume abcdefghijk 2018-06-29 07:09:43,451 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [ { "owner" : { "name" : "root" }, "quota" : { "unit" : "TB", "size" : 1048576 }, "volumeName" : "nnvolume1", "createdOn" : "Sun, 18 Sep +50444 15:12:11 GMT", "createdBy" : "root" }, { "owner" : { "name" : "root" }, "quota" : { "unit" : "TB", "size" : 1048576 }, "volumeName" : "nnvolume2", "createdOn" : "Tue, 27 Sep +50444 13:05:43 GMT", "createdBy" : "root" } ] {noformat} expectation: It should not accept random values as argument -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-208) ozone createVolume command ignores the first character of the "volume name" given as argument
Nilotpal Nandi created HDDS-208: --- Summary: ozone createVolume command ignores the first character of the "volume name" given as argument Key: HDDS-208 URL: https://issues.apache.org/jira/browse/HDDS-208 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi Fix For: 0.2.1 createVolume command ran to create volume "testvolume123". Volume created with name "estvolume123" instead of "testvolume123". It ignores the first character of the volume name {noformat} [root@ozone-vm bin]# ./ozone oz -createVolume testvolume123 -user root 2018-07-02 05:33:35,510 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-07-02 05:33:36,093 [main] INFO - Creating Volume: estvolume123, with root as owner and quota set to 1152921504606846976 bytes. {noformat} ozone listVolume command : {noformat} [root@ozone-vm bin]# ./ozone oz -listVolume / 2018-07-02 05:36:47,835 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [ { "owner" : { "name" : "root" }, "quota" : { "unit" : "TB", "size" : 1048576 }, "volumeName" : "nnvolume1", "createdOn" : "Sun, 18 Sep +50444 15:12:11 GMT", "createdBy" : "root" .. .. }, { "owner" : { "name" : "root" }, "quota" : { "unit" : "TB", "size" : 1048576 }, "volumeName" : "estvolume123", "createdOn" : "Sat, 17 May +50470 08:01:41 GMT", "createdBy" : "root" } ] {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-209) createVolume command throws error when user is not present locally but creates the volume
Nilotpal Nandi created HDDS-209: --- Summary: createVolume command throws error when user is not present locally but creates the volume Key: HDDS-209 URL: https://issues.apache.org/jira/browse/HDDS-209 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi Fix For: 0.2.1 user "test_user3" does not exist locally. When -createVolume command is ran for the user "test_user3", it throws error on standard output but successfully creates the volume. The exit code for the command execution is non-zero. {noformat} [root@ozone-vm bin]# ./ozone oz -createVolume /testvolume121 -user test_user3 2018-07-02 06:01:37,020 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-07-02 06:01:37,605 WARN security.ShellBasedUnixGroupsMapping: unable to return groups for user test_user3 PartialGroupNameException The user name 'test_user3' is not found. id: test_user3: no such user id: test_user3: no such user at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:294) at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:207) at org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroups(ShellBasedUnixGroupsMapping.java:97) at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.getGroups(JniBasedUnixGroupsMappingWithFallback.java:51) at org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupList(Groups.java:384) at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:319) at org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:269) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) at com.google.common.cache.LocalCache.get(LocalCache.java:3965) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829) at org.apache.hadoop.security.Groups.getGroups(Groups.java:227) at org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1547) at org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1535) at org.apache.hadoop.ozone.client.rpc.RpcClient.createVolume(RpcClient.java:190) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ozone.client.OzoneClientInvocationHandler.invoke(OzoneClientInvocationHandler.java:54) at com.sun.proxy.$Proxy11.createVolume(Unknown Source) at org.apache.hadoop.ozone.client.ObjectStore.createVolume(ObjectStore.java:77) at org.apache.hadoop.ozone.web.ozShell.volume.CreateVolumeHandler.execute(CreateVolumeHandler.java:98) at org.apache.hadoop.ozone.web.ozShell.Shell.dispatch(Shell.java:395) at org.apache.hadoop.ozone.web.ozShell.Shell.run(Shell.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90) at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:114) 2018-07-02 06:01:37,611 [main] INFO - Creating Volume: testvolume121, with test_user3 as owner and quota set to 1152921504606846976 bytes. {noformat} {noformat} [root@ozone-vm bin]# ./ozone oz -listVolume / -user test_user3 2018-07-02 06:02:20,385 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [ { "owner" : { "name" : "test_user3" }, "quota" : { "unit" : "TB", "size" : 1048576 }, "volumeName" : "testvolume121", "createdOn" : "Thu, 05 Jun +50470 19:07:00 GMT", "createdBy" : "test_user3" } ] {noformat} Expectation : -- Volume should not present if local user is not present. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-210) ozone getKey command always expects the filename to be present along with file-path in "-file" argument
Nilotpal Nandi created HDDS-210: --- Summary: ozone getKey command always expects the filename to be present along with file-path in "-file" argument Key: HDDS-210 URL: https://issues.apache.org/jira/browse/HDDS-210 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Environment: ozone getKey command always expects the filename to be present along with the file-path for the "-file" argument. It throws error if filename is not provided. {noformat} [root@ozone-vm bin]# ./ozone oz -getKey /nnvolume1/bucket123/passwd -file /test1/ 2018-07-02 06:45:27,355 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Command Failed : {"httpCode":0,"shortMessage":"/test1/exists. Download will overwrite an existing file. Aborting.","resource":null,"message":"/test1/exists. Download will overwrite an existing file. Aborting.","requestID":null,"hostName":null} [root@ozone-vm bin]# ./ozone oz -getKey /nnvolume1/bucket123/passwd -file /test1/passwd 2018-07-02 06:45:39,722 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-07-02 06:45:40,354 INFO conf.ConfUtils: raft.rpc.type = GRPC (default) 2018-07-02 06:45:40,366 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-07-02 06:45:40,372 INFO conf.ConfUtils: raft.client.rpc.retryInterval = 300 ms (default) 2018-07-02 06:45:40,374 INFO conf.ConfUtils: raft.client.async.outstanding-requests.max = 100 (default) 2018-07-02 06:45:40,374 INFO conf.ConfUtils: raft.client.async.scheduler-threads = 3 (default) 2018-07-02 06:45:40,507 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-07-02 06:45:40,507 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-07-02 06:45:40,814 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms (default){noformat} Expectation : -- ozone getKey should work even when only file-path is provided (without filename). It should create a file in the given file-path with its key's name as its name. i.e, given , /test1 is a directory . if ./ozone oz -getKey /nnvolume1/bucket123/passwd -file /test1 is run, file 'passwd' should be created in the directory /test1 . Reporter: Nilotpal Nandi Fix For: 0.2.1 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-264) 'oz' subcommand reference is not present in 'ozone' command help
Nilotpal Nandi created HDDS-264: --- Summary: 'oz' subcommand reference is not present in 'ozone' command help Key: HDDS-264 URL: https://issues.apache.org/jira/browse/HDDS-264 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi Fix For: 0.2.1 'oz' subcommand is not present in ozone help. ozone help: {noformat} hadoop@8ceb8dfccb36:~/bin$ ./ozone Usage: ozone [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS] OPTIONS is none or any of: --buildpaths attempt to add class files from build tree --config dir Hadoop config directory --daemon (start|status|stop) operate on a daemon --debug turn on shell script debug mode --help usage information --hostnames list[,of,host,names] hosts to use in worker mode --hosts filename list of hosts to use in worker mode --loglevel level set the log4j level for this command --workers turn on worker mode SUBCOMMAND is one of: Admin Commands: jmxget get JMX exported values from NameNode or DataNode. Client Commands: classpath prints the class path needed to get the hadoop jar and the required libraries envvars display computed Hadoop environment variables freon runs an ozone data generator genconf generate minimally required ozone configs and output to ozone-site.xml in specified path genesis runs a collection of ozone benchmarks to help with tuning. getozoneconf get ozone config values from configuration noz ozone debug tool, convert ozone metadata into relational data o3 command line interface for ozone scmcli run the CLI of the Storage Container Manager version print the version Daemon Commands: datanode run a HDDS datanode om Ozone Manager scm run the Storage Container Manager service SUBCOMMAND may print help when invoked w/o parameters or with -h. {noformat} 'oz' subcommand example : {noformat} hadoop@8ceb8dfccb36:~/bin$ ./ozone oz -listVolume / 2018-07-19 14:51:25 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [ { "owner" : { "name" : "hadoop" }, "quota" : { "unit" : "TB", "size" : 1048576 }, "volumeName" : "vol-0-01597", "createdOn" : "Sat, 20 Feb +50517 10:11:35 GMT", "createdBy" : "hadoop" }, { "owner" : { "name" : "hadoop" }, "quota" : { "unit" : "TB", "size" : 1048576 }, "volumeName" : "vol-0-19478", "createdOn" : "Thu, 03 Jun +50517 22:23:12 GMT", "createdBy" : "hadoop" }, { "owner" : { "name" : "hadoop" }, "quota" : { "unit" : "TB", "size" : 1048576 } {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-275) DeleteVolume command does not give any confirmation message on successful execution
Nilotpal Nandi created HDDS-275: --- Summary: DeleteVolume command does not give any confirmation message on successful execution Key: HDDS-275 URL: https://issues.apache.org/jira/browse/HDDS-275 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi Fix For: 0.2.1 On deleting a Volume and successful execution , there is no confirmation message saying that the "volume is deleted". For example : {noformat} hadoop@d4eb7d83cdee:~/bin$ for i in $(seq 201 1000) > do > ./ozone oz -deleteVolume /vol1${i} > done 2018-07-20 09:11:50 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-07-20 09:11:52 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable .. .. .. {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-281) Need container size distribution metric in OzoneManager UI
Nilotpal Nandi created HDDS-281: --- Summary: Need container size distribution metric in OzoneManager UI Key: HDDS-281 URL: https://issues.apache.org/jira/browse/HDDS-281 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Manager Reporter: Nilotpal Nandi It would be good if we have some metric/histogram in OzoneManager UI indicating the different container size range and corresponding percentages for the same created in the cluster. For example : 0-2 GB 10% 2-4 GB . 20% 4-5 GB 70% 5+ GB 0% -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-283) Need an option to list all volumes created in the cluster
Nilotpal Nandi created HDDS-283: --- Summary: Need an option to list all volumes created in the cluster Key: HDDS-283 URL: https://issues.apache.org/jira/browse/HDDS-283 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi Fix For: 0.2.1 Currently , listVolume command either gives : 1) all the volumes created by a particular user , using -user argument. 2) or , all the volumes created by the logged in user , if no -user argument is provided. We need an option to list all the volumes created in the cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-301) ozone does not contain subcommand to run ozoneFS commands
Nilotpal Nandi created HDDS-301: --- Summary: ozone does not contain subcommand to run ozoneFS commands Key: HDDS-301 URL: https://issues.apache.org/jira/browse/HDDS-301 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Filesystem Reporter: Nilotpal Nandi Fix For: 0.2.1 Unlike hdfs , currently ozone does not have the sub-command to run ozoneFS commands. In hdfs, we have dfs subcommand to run hdfs filesystem commands. Similar, subcommand needs to be added for ozone -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-314) ozoneShell putKey command overwrites the existing key having same name
Nilotpal Nandi created HDDS-314: --- Summary: ozoneShell putKey command overwrites the existing key having same name Key: HDDS-314 URL: https://issues.apache.org/jira/browse/HDDS-314 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi Fix For: 0.2.1 steps taken : 1) created a volume root-volume and a bucket root-bucket. 2) Ran following command to put a key with name 'passwd' {noformat} hadoop@08315aa4b367:~/bin$ ./ozone oz -putKey /root-volume/root-bucket/passwd -file /etc/services -v 2018-08-02 09:20:17 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Volume Name : root-volume Bucket Name : root-bucket Key Name : passwd File Hash : 567c100888518c1163b3462993de7d47 2018-08-02 09:20:18 INFO ConfUtils:41 - raft.rpc.type = GRPC (default) 2018-08-02 09:20:18 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-02 09:20:18 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300 ms (default) 2018-08-02 09:20:18 INFO ConfUtils:41 - raft.client.async.outstanding-requests.max = 100 (default) 2018-08-02 09:20:18 INFO ConfUtils:41 - raft.client.async.scheduler-threads = 3 (default) 2018-08-02 09:20:18 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-08-02 09:20:18 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-02 09:20:18 INFO ConfUtils:41 - raft.client.rpc.request.timeout = 3000 ms (default) Aug 02, 2018 9:20:18 AM org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy {noformat} 3) Ran following command to put a key with name 'passwd' again. {noformat} hadoop@08315aa4b367:~/bin$ ./ozone oz -putKey /root-volume/root-bucket/passwd -file /etc/passwd -v 2018-08-02 09:20:41 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Volume Name : root-volume Bucket Name : root-bucket Key Name : passwd File Hash : b056233571cc80d6879212911cb8e500 2018-08-02 09:20:41 INFO ConfUtils:41 - raft.rpc.type = GRPC (default) 2018-08-02 09:20:42 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-02 09:20:42 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300 ms (default) 2018-08-02 09:20:42 INFO ConfUtils:41 - raft.client.async.outstanding-requests.max = 100 (default) 2018-08-02 09:20:42 INFO ConfUtils:41 - raft.client.async.scheduler-threads = 3 (default) 2018-08-02 09:20:42 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-08-02 09:20:42 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-02 09:20:42 INFO ConfUtils:41 - raft.client.rpc.request.timeout = 3000 ms (default) Aug 02, 2018 9:20:42 AM org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy{noformat} key 'passwd' was overwritten with new content and it did not throw any saying that the key is already present. Expectation : --- key overwrite with same name should not be allowed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-315) ozoneShell infoKey does not work for directories created as key and throws 'KEY_NOT_FOUND' error
Nilotpal Nandi created HDDS-315: --- Summary: ozoneShell infoKey does not work for directories created as key and throws 'KEY_NOT_FOUND' error Key: HDDS-315 URL: https://issues.apache.org/jira/browse/HDDS-315 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi Fix For: 0.2.1 infoKey for directories created using ozoneFs does not work and throws 'KEY_NOT_FOUND' error. However, it shows up in the 'listKey' command. Here in this example, 'dir1' was created using ozoneFS , infoKey for the directory throws error. {noformat} hadoop@08315aa4b367:~/bin./ozone oz -infoKey /root-volume/root-bucket/dir1 2018-08-02 11:34:06 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Command Failed : Lookup key failed, error:KEY_NOT_FOUND hadoop@08315aa4b367:~/bin$ ./ozone oz -infoKey /root-volume/root-bucket/dir1/ 2018-08-02 11:34:16 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Command Failed : Lookup key failed, error:KEY_NOT_FOUND hadoop@08315aa4b367:~/bin$ ./ozone oz -listKey /root-volume/root-bucket/ 2018-08-02 11:34:21 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [ { "version" : 0, "md5hash" : null, "createdOn" : "Wed, 07 May +50555 12:44:16 GMT", "modifiedOn" : "Wed, 07 May +50555 12:44:30 GMT", "size" : 0, "keyName" : "dir1/" }, { "version" : 0, "md5hash" : null, "createdOn" : "Wed, 07 May +50555 14:14:06 GMT", "modifiedOn" : "Wed, 07 May +50555 14:14:19 GMT", "size" : 0, "keyName" : "dir2/" }, { "version" : 0, "md5hash" : null, "createdOn" : "Thu, 08 May +50555 21:40:55 GMT", "modifiedOn" : "Thu, 08 May +50555 21:40:59 GMT", "size" : 0, "keyName" : "dir2/b1/"{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-318) ratis INFO logs should not shown during ozoneFs command-line execution
Nilotpal Nandi created HDDS-318: --- Summary: ratis INFO logs should not shown during ozoneFs command-line execution Key: HDDS-318 URL: https://issues.apache.org/jira/browse/HDDS-318 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Filesystem Reporter: Nilotpal Nandi Fix For: 0.2.1 ratis INFOs should not be shown during ozoneFS CLI execution. Please find the snippet from one othe execution : {noformat} hadoop@08315aa4b367:~/bin$ ./ozone fs -put /etc/passwd /p2 2018-08-02 12:17:18 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-08-02 12:17:19 INFO ConfUtils:41 - raft.rpc.type = GRPC (default) 2018-08-02 12:17:19 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-02 12:17:19 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300 ms (default) 2018-08-02 12:17:19 INFO ConfUtils:41 - raft.client.async.outstanding-requests.max = 100 (default) 2018-08-02 12:17:19 INFO ConfUtils:41 - raft.client.async.scheduler-threads = 3 (default) 2018-08-02 12:17:19 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-08-02 12:17:19 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-02 12:17:20 INFO ConfUtils:41 - raft.client.rpc.request.timeout = 3000 ms (default) Aug 02, 2018 12:17:20 PM org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy WARNING: Failed to construct URI for proxy lookup, proceeding without proxy .. .. .. {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-321) ozoneFS put/copyFromLocal command does not work for a directory when the directory contains file(s) as well as subdirectories
Nilotpal Nandi created HDDS-321: --- Summary: ozoneFS put/copyFromLocal command does not work for a directory when the directory contains file(s) as well as subdirectories Key: HDDS-321 URL: https://issues.apache.org/jira/browse/HDDS-321 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi Fix For: 0.2.1 Steps taken : - # Created a local directory 'TEST_DIR1' which contains directory "SUB_DIR1" and a file "test_file1". # Ran "./ozone fs -put TEST_DIR1/ /" . The command kept on running , throwing error on console. stack trace of the error thrown on the console : {noformat} 2018-08-02 12:55:46 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-08-02 12:55:46 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-02 12:55:46 INFO ConfUtils:41 - raft.client.rpc.request.timeout = 3000 ms (default) Aug 02, 2018 12:55:46 PM org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy WARNING: Failed to construct URI for proxy lookup, proceeding without proxy java.net.URISyntaxException: Illegal character in hostname at index 13: https://ozone_datanode_3.ozone_default:9858 at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.parseHostname(URI.java:3387) at java.net.URI$Parser.parseServer(URI.java:3236) at java.net.URI$Parser.parseAuthority(URI.java:3155) at java.net.URI$Parser.parseHierarchical(URI.java:3097) at java.net.URI$Parser.parse(URI.java:3053) at java.net.URI.(URI.java:673) at org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128) at org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118) at org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207) at org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130) at org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032) at org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$4.get(ManagedChannelImpl.java:403) at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:238) at org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:386) at org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:679) at org.apache.ratis.shaded.io.grpc.stub.ClientCalls.startCall(ClientCalls.java:293) at org.apache.ratis.shaded.io.grpc.stub.ClientCalls.asyncStreamingRequestCall(ClientCalls.java:283) at org.apache.ratis.shaded.io.grpc.stub.ClientCalls.asyncBidiStreamingCall(ClientCalls.java:92) at org.apache.ratis.shaded.proto.grpc.RaftClientProtocolServiceGrpc$RaftClientProtocolServiceStub.append(RaftClientProtocolServiceGrpc.java:208) at org.apache.ratis.grpc.client.RaftClientProtocolClient.appendWithTimeout(RaftClientProtocolClient.java:139) at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:109) at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:88) at org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:302) at org.apache.ratis.client.impl.RaftClientImpl.sendRequestWithRetry(RaftClientImpl.java:256) at org.apache.ratis.client.impl.RaftClientImpl.send(RaftClientImpl.java:192) at org.apache.ratis.client.impl.RaftClientImpl.send(RaftClientImpl.java:173) at org.apache.ratis.client.RaftClient.send(RaftClient.java:80) at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequest(XceiverClientRatis.java:218) at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommand(XceiverClientRatis.java:235) at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunk(ContainerProtocolCalls.java:219) at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.writeChunkToContainer(ChunkOutputStream.java:220) at org.apache.hadoop.hdds.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:150) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:486) at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:326) at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:57) at org.apache.hadoop.fs.FSDataO
[jira] [Created] (HDDS-337) keys created with key name having special character/wildcard should not allowed
Nilotpal Nandi created HDDS-337: --- Summary: keys created with key name having special character/wildcard should not allowed Key: HDDS-337 URL: https://issues.apache.org/jira/browse/HDDS-337 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi Fix For: 0.2.1 Please find the snippet of command execution. Here , the keys are created with wildcard special character in its key name. Expectation : wildcard special characters should not be allowed. {noformat} hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/d++ -file /etc/services -v 2018-08-08 13:17:48 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Volume Name : root-volume Bucket Name : root-bucket Key Name : d++ File Hash : 567c100888518c1163b3462993de7d47 Key Name : d++ does not exist, creating it 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.rpc.type = GRPC (default) 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300 ms (default) 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.client.async.outstanding-requests.max = 100 (default) 2018-08-08 13:17:48 INFO ConfUtils:41 - raft.client.async.scheduler-threads = 3 (default) 2018-08-08 13:17:49 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-08-08 13:17:49 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-08 13:17:49 INFO ConfUtils:41 - raft.client.rpc.request.timeout = 3000 ms (default) Aug 08, 2018 1:17:49 PM org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy WARNING: Failed to construct URI for proxy lookup, proceeding without proxy java.net.URISyntaxException: Illegal character in hostname at index 13: https://ozone_datanode_1.ozone_default:9858 at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.parseHostname(URI.java:3387) at java.net.URI$Parser.parseServer(URI.java:3236) at java.net.URI$Parser.parseAuthority(URI.java:3155) at java.net.URI$Parser.parseHierarchical(URI.java:3097) at java.net.URI$Parser.parse(URI.java:3053) at java.net.URI.(URI.java:673) at org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128) at org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118) at org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207) at org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130) at org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032) at org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:1000) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:1044) at org.apache.ratis.shaded.io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/d** -file /etc/passwd -v 2018-08-08 13:18:13 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Volume Name : root-volume Bucket Name : root-bucket Key Name : d** File Hash : b056233571cc80d6879212911cb8e500 Key Name : d** does not exist, creating it 2018-08-08 13:18:14 INFO ConfUtils:41 - raft.rpc.type = GRPC (default) 2018-08-08 13:18:14 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-08 13:18:14 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300 ms (default) 2018-08-08 13:18:14 INFO ConfUtils:41 - raft.client.async.outstanding-requests.max = 100 (default) 2018-08-08 13:18:14 INFO ConfUtils:41 - raft.client.async.scheduler-threads = 3 (default) 2018-08-08 13:18:14 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-08-08 13:18:14 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-08 13:18:14 INFO ConfUtils:41 - raft.client.rpc.request.timeout
[jira] [Created] (HDDS-338) ozoneFS allows to create file key and directory key with same keyname
Nilotpal Nandi created HDDS-338: --- Summary: ozoneFS allows to create file key and directory key with same keyname Key: HDDS-338 URL: https://issues.apache.org/jira/browse/HDDS-338 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Filesystem Reporter: Nilotpal Nandi Fix For: 0.2.1 steps taken : -- 1. created a directory through ozoneFS interface. {noformat} hadoop@1a1fa8a11332:~/bin$ ./ozone fs -mkdir /temp1/ 2018-08-08 13:50:26 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable hadoop@1a1fa8a11332:~/bin$ ./ozone fs -ls / 2018-08-08 14:09:59 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items drwxrwxrwx - 0 2018-08-08 13:51 /temp1{noformat} 2. create a new key with name 'temp1' at same bucket. {noformat} hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/temp1 -file /etc/passwd 2018-08-08 14:10:34 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.rpc.type = GRPC (default) 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300 ms (default) 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.async.outstanding-requests.max = 100 (default) 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.async.scheduler-threads = 3 (default) 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 (custom) 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.request.timeout = 3000 ms (default) Aug 08, 2018 2:10:36 PM org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy WARNING: Failed to construct URI for proxy lookup, proceeding without proxy java.net.URISyntaxException: Illegal character in hostname at index 13: https://ozone_datanode_3.ozone_default:9858 at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.parseHostname(URI.java:3387) at java.net.URI$Parser.parseServer(URI.java:3236) at java.net.URI$Parser.parseAuthority(URI.java:3155) at java.net.URI$Parser.parseHierarchical(URI.java:3097) at java.net.URI$Parser.parse(URI.java:3053) at java.net.URI.(URI.java:673) at org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128) at org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118) at org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207) at org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130) at org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032) at org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:1000) at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:1044) at org.apache.ratis.shaded.io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748){noformat} Observed that there are multiple entries of 'temp1' when ozone fs -ls command is run. Also . both the entries are considered as file . '/temp1' directory is not visible anymore. {noformat} hadoop@1a1fa8a11332:~/bin$ ./ozone fs -ls / 2018-08-08 14:10:41 WARN NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 2 items -rw-rw-rw- 1 963 2018-08-08 14:10 /temp1 -rw-rw-rw- 1 963 2018-08-08 14:10 /temp1{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-403) infoKey shows wrong "createdOn", "modifiedOn" metadata for key
Nilotpal Nandi created HDDS-403: --- Summary: infoKey shows wrong "createdOn", "modifiedOn" metadata for key Key: HDDS-403 URL: https://issues.apache.org/jira/browse/HDDS-403 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Reporter: Nilotpal Nandi Fix For: 0.2.1 1. ran putKey command for a file {noformat} [root@ctr-e138-1518143905142-459606-01-03 bin]# ./ozone oz -putKey /test-vol1/test-bucket1/file1 -file /etc/passwd -v 2018-09-05 10:25:11,498 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Volume Name : test-vol1 Bucket Name : test-bucket1 Key Name : file1 File Hash : 8164cc3d5b05c44b73a6277661aa4645 2018-09-05 10:25:12,377 INFO conf.ConfUtils: raft.rpc.type = GRPC (default) 2018-09-05 10:25:12,390 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-09-05 10:25:12,402 INFO conf.ConfUtils: raft.client.rpc.retryInterval = 300 ms (default) 2018-09-05 10:25:12,407 INFO conf.ConfUtils: raft.client.async.outstanding-requests.max = 100 (default) 2018-09-05 10:25:12,407 INFO conf.ConfUtils: raft.client.async.scheduler-threads = 3 (default) 2018-09-05 10:25:12,518 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-09-05 10:25:12,518 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-09-05 10:25:12,866 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms (default) 2018-09-05 10:25:13,644 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-09-05 10:25:13,644 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-09-05 10:25:13,645 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms (default) [root@ctr-e138-1518143905142-459606-01-03 bin]# ./ozone oz -getKey /test-vol1/test-bucket1/file1 -file getkey3 2018-09-05 10:25:22,020 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-09-05 10:25:22,778 INFO conf.ConfUtils: raft.rpc.type = GRPC (default) 2018-09-05 10:25:22,790 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-09-05 10:25:22,800 INFO conf.ConfUtils: raft.client.rpc.retryInterval = 300 ms (default) 2018-09-05 10:25:22,804 INFO conf.ConfUtils: raft.client.async.outstanding-requests.max = 100 (default) 2018-09-05 10:25:22,805 INFO conf.ConfUtils: raft.client.async.scheduler-threads = 3 (default) 2018-09-05 10:25:22,890 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-09-05 10:25:22,890 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-09-05 10:25:23,250 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms (default) 2018-09-05 10:25:24,066 INFO conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-09-05 10:25:24,067 INFO conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-09-05 10:25:24,067 INFO conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms (default){noformat} 2. Ran infoKey on that key {noformat} [root@ctr-e138-1518143905142-459606-01-03 bin]# ./ozone oz -infoKey /test-vol1/test-bucket1/file1 -v 2018-09-05 10:54:42,053 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Volume Name : test-vol1 Bucket Name : test-bucket1 Key Name : file1 { "version" : 0, "md5hash" : null, "createdOn" : "Sat, 14 Dec +114522267 00:51:17 GMT", "modifiedOn" : "Fri, 09 Jun +50648 04:30:12 GMT", "size" : 4659, "keyName" : "file1", "keyLocations" : [ { "containerID" : 16, "localID" : 1536143112267, "length" : 4659, "offset" : 0 } ] }{noformat} "createdOn" and "modifiedOn" metadata are incorrect. Here is the current date: {noformat} [root@ctr-e138-1518143905142-459606-01-03 bin]# date Wed Sep 5 10:54:52 UTC 2018{noformat} Also , the "md5hash" for the key is showing as null. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-407) ozone logs are wriiten to ozone.log. instead of ozone.log
Nilotpal Nandi created HDDS-407: --- Summary: ozone logs are wriiten to ozone.log. instead of ozone.log Key: HDDS-407 URL: https://issues.apache.org/jira/browse/HDDS-407 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Nilotpal Nandi Fix For: 0.2.1 Please refer below details ozone related logs are written to ozone.log.2018-09-05 instead of ozone.log. Also, please check the timestamps of the logs. The cluster was created {noformat} [root@ctr-e138-1518143905142-459606-01-02 logs]# ls -lhart /root/hadoop_trunk/ozone-0.2.1-SNAPSHOT/logs/ total 968K drwxr-xr-x 9 root root 4.0K Sep 5 10:04 .. -rw-r--r-- 1 root root 0 Sep 5 10:04 fairscheduler-statedump.log -rw-r--r-- 1 root root 17K Sep 5 10:05 hadoop-root-om-ctr-e138-1518143905142-459606-01-02.hwx.site.out.1 -rw-r--r-- 1 root root 16K Sep 5 10:10 hadoop-root-om-ctr-e138-1518143905142-459606-01-02.hwx.site.out -rw-r--r-- 1 root root 11K Sep 5 10:10 hadoop-root-om-ctr-e138-1518143905142-459606-01-02.hwx.site.log -rw-r--r-- 1 root root 17K Sep 6 05:42 hadoop-root-datanode-ctr-e138-1518143905142-459606-01-02.hwx.site.out -rw-r--r-- 1 root root 2.1K Sep 6 13:20 ozone.log -rw-r--r-- 1 root root 67K Sep 6 13:22 hadoop-root-datanode-ctr-e138-1518143905142-459606-01-02.hwx.site.log drwxr-xr-x 2 root root 4.0K Sep 6 13:31 . -rw-r--r-- 1 root root 811K Sep 6 13:39 ozone.log.2018-09-05 [root@ctr-e138-1518143905142-459606-01-02 logs]# date Thu Sep 6 13:39:47 UTC 2018{noformat} tail of ozone.log {noformat} [root@ctr-e138-1518143905142-459606-01-02 logs]# tail -f ozone.log 2018-09-06 10:51:56,616 [IPC Server handler 13 on 9889] DEBUG (KeyManagerImpl.java:255) - Key 0file allocated in volume test-vol2 bucket test-bucket2 2018-09-06 10:52:18,570 [IPC Server handler 9 on 9889] DEBUG (KeyManagerImpl.java:255) - Key 0file1 allocated in volume test-vol2 bucket test-bucket2 2018-09-06 10:52:32,256 [IPC Server handler 12 on 9889] DEBUG (KeyManagerImpl.java:255) - Key 0file2 allocated in volume test-vol2 bucket test-bucket2 2018-09-06 10:53:11,008 [IPC Server handler 14 on 9889] DEBUG (KeyManagerImpl.java:255) - Key 0file2 allocated in volume test-vol2 bucket test-bucket2 2018-09-06 10:53:28,316 [IPC Server handler 10 on 9889] DEBUG (KeyManagerImpl.java:255) - Key 0file2 allocated in volume test-vol2 bucket test-bucket2 2018-09-06 10:53:39,509 [IPC Server handler 17 on 9889] DEBUG (KeyManagerImpl.java:255) - Key 0file3 allocated in volume test-vol2 bucket test-bucket2 2018-09-06 11:31:02,388 [IPC Server handler 19 on 9889] DEBUG (KeyManagerImpl.java:255) - Key 2GBFILE allocated in volume test-vol2 bucket test-bucket2 2018-09-06 11:32:44,269 [IPC Server handler 12 on 9889] DEBUG (KeyManagerImpl.java:255) - Key 2GBFILE_1 allocated in volume test-vol2 bucket test-bucket2 2018-09-06 13:17:33,408 [IPC Server handler 16 on 9889] DEBUG (KeyManagerImpl.java:255) - Key FILEWITHZEROS allocated in volume test-vol2 bucket test-bucket2 2018-09-06 13:20:13,897 [IPC Server handler 15 on 9889] DEBUG (KeyManagerImpl.java:255) - Key FILEWITHZEROS1 allocated in volume test-vol2 bucket test-bucket2{noformat} tail of ozone.log.2018-09-05: {noformat} root@ctr-e138-1518143905142-459606-01-02 logs]# tail -50 ozone.log.2018-09-05 2018-09-06 13:28:57,866 [BlockDeletingService#8] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 13:29:07,816 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3266 2018-09-06 13:29:13,687 [Datanode ReportManager Thread - 0] DEBUG (ContainerSet.java:191) - Starting container report iteration. 2018-09-06 13:29:37,816 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3267 2018-09-06 13:29:57,866 [BlockDeletingService#8] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 13:30:07,816 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3268 2018-09-06 13:30:19,186 [Datanode ReportManager Thread - 0] DEBUG (ContainerSet.java:191) - Starting container report iteration. 2018-09-06 13:30:37,816 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3269 2018-09-06 13:30:57,866 [BlockDeletingService#8] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 13:31:07,816 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3270
[jira] [Created] (HDDS-408) Read (getKey) operation is very slow
Nilotpal Nandi created HDDS-408: --- Summary: Read (getKey) operation is very slow Key: HDDS-408 URL: https://issues.apache.org/jira/browse/HDDS-408 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode, Ozone Manager Reporter: Nilotpal Nandi Fix For: 0.2.1 PutKey operation for a 2GB file took around ~42 secs . real 0m41.955s user 0m19.367s sys 0m7.480s For the same Key, getKey operation took around 8min 49 secs. real 0m41.955s user 0m19.367s sys 0m7.480s Repeated operation multiple times, results were similar. For read operation , the client waits for a long time. In ozone.log, there is a long trail of these messages on all datanodes : {noformat} 2018-09-06 14:10:52,288 [BlockDeletingService#2] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 14:11:02,413 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3350 2018-09-06 14:11:32,414 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3351 2018-09-06 14:11:41,797 [Datanode ReportManager Thread - 0] DEBUG (ContainerSet.java:191) - Starting container report iteration. 2018-09-06 14:11:52,288 [BlockDeletingService#2] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 14:12:02,414 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3352 2018-09-06 14:12:32,416 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3353 2018-09-06 14:12:48,723 [Datanode ReportManager Thread - 0] DEBUG (ContainerSet.java:191) - Starting container report iteration. 2018-09-06 14:12:52,289 [BlockDeletingService#2] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 14:13:02,416 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3354 2018-09-06 14:13:32,416 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3355 2018-09-06 14:13:52,289 [BlockDeletingService#2] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 14:14:02,416 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3356 2018-09-06 14:14:31,725 [Datanode ReportManager Thread - 2] DEBUG (ContainerSet.java:191) - Starting container report iteration. 2018-09-06 14:14:32,416 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3357 2018-09-06 14:14:52,290 [BlockDeletingService#2] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 14:15:02,417 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3358 2018-09-06 14:15:32,418 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3359 2018-09-06 14:15:52,290 [BlockDeletingService#2] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 14:16:02,419 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3360 2018-09-06 14:16:26,897 [Datanode ReportManager Thread - 0] DEBUG (ContainerSet.java:191) - Starting container report iteration. 2018-09-06 14:16:32,420 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3361 2018-09-06 14:16:52,290 [BlockDeletingService#2] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-06 14:17:02,420 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3362 2018-09-06 14:17:32,420 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:145) - Executing cycle Number : 3363 2018-09-06 14:17:48,024 [Datanode ReportManager Thread - 0] DEBUG (ContainerSet.java:191) - Starting container report iteration. 2018-09-06 14:17:52,291 [BlockDeletingService#2] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending de
[jira] [Created] (HDDS-410) ozone scmcli is not working properly
Nilotpal Nandi created HDDS-410: --- Summary: ozone scmcli is not working properly Key: HDDS-410 URL: https://issues.apache.org/jira/browse/HDDS-410 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Reporter: Nilotpal Nandi Fix For: 0.2.1 On running ozone scmcli for a container ID, it gives the following output : {noformat} [root@ctr-e138-1518143905142-459606-01-02 bin]# ./ozone scmcli list --start=17 Infinite recursion (StackOverflowError) (through reference chain: org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSupplier["initializer"]->org.apache.ratis.protocol.RaftId$$Lambda$17/1523553211["arg$1"]->org.apache.ratis.protocol.RaftGroupId["uuidString"]->org.apache.ratis.util.MemoizedSup
[jira] [Resolved] (HDDS-321) ozoneFS put/copyFromLocal command does not work for a directory when the directory contains file(s) as well as subdirectories
[ https://issues.apache.org/jira/browse/HDDS-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nilotpal Nandi resolved HDDS-321. - Resolution: Fixed > ozoneFS put/copyFromLocal command does not work for a directory when the > directory contains file(s) as well as subdirectories > - > > Key: HDDS-321 > URL: https://issues.apache.org/jira/browse/HDDS-321 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Nilotpal Nandi >Assignee: Nilotpal Nandi >Priority: Blocker > Fix For: 0.2.1 > > > Steps taken : > - > # Created a local directory 'TEST_DIR1' which contains directory "SUB_DIR1" > and a file "test_file1". > # Ran "./ozone fs -put TEST_DIR1/ /" . The command kept on running , > throwing error on console. > stack trace of the error thrown on the console : > {noformat} > 2018-08-02 12:55:46 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB > (=1048576) (default) > 2018-08-02 12:55:46 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432 > (custom) > 2018-08-02 12:55:46 INFO ConfUtils:41 - raft.client.rpc.request.timeout = > 3000 ms (default) > Aug 02, 2018 12:55:46 PM > org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy > WARNING: Failed to construct URI for proxy lookup, proceeding without proxy > java.net.URISyntaxException: Illegal character in hostname at index 13: > https://ozone_datanode_3.ozone_default:9858 > at java.net.URI$Parser.fail(URI.java:2848) > at java.net.URI$Parser.parseHostname(URI.java:3387) > at java.net.URI$Parser.parseServer(URI.java:3236) > at java.net.URI$Parser.parseAuthority(URI.java:3155) > at java.net.URI$Parser.parseHierarchical(URI.java:3097) > at java.net.URI$Parser.parse(URI.java:3053) > at java.net.URI.(URI.java:673) > at > org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128) > at > org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118) > at > org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207) > at > org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188) > at > org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130) > at > org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79) > at > org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032) > at > org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73) > at > org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$4.get(ManagedChannelImpl.java:403) > at > org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:238) > at > org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:386) > at > org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:679) > at > org.apache.ratis.shaded.io.grpc.stub.ClientCalls.startCall(ClientCalls.java:293) > at > org.apache.ratis.shaded.io.grpc.stub.ClientCalls.asyncStreamingRequestCall(ClientCalls.java:283) > at > org.apache.ratis.shaded.io.grpc.stub.ClientCalls.asyncBidiStreamingCall(ClientCalls.java:92) > at > org.apache.ratis.shaded.proto.grpc.RaftClientProtocolServiceGrpc$RaftClientProtocolServiceStub.append(RaftClientProtocolServiceGrpc.java:208) > at > org.apache.ratis.grpc.client.RaftClientProtocolClient.appendWithTimeout(RaftClientProtocolClient.java:139) > at > org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:109) > at > org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:88) > at > org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:302) > at > org.apache.ratis.client.impl.RaftClientImpl.sendRequestWithRetry(RaftClientImpl.java:256) > at org.apache.ratis.client.impl.RaftClientImpl.send(RaftClientImpl.java:192) > at org.apache.ratis.client.impl.RaftClientImpl.send(RaftClientImpl.java:173) > at org.apache.ratis.client.RaftClient.send(RaftClient.java:80) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequest(XceiverClientRatis.java:218) > at > org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommand(XceiverClientRatis.java:235) > at > org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunk(ContainerProtocolCalls.java:219) > at > org.apache.hadoop.hdds.scm.storage.ChunkOutputS
[jira] [Created] (HDDS-420) putKey failing with KEY_ALLOCATION_ERROR
Nilotpal Nandi created HDDS-420: --- Summary: putKey failing with KEY_ALLOCATION_ERROR Key: HDDS-420 URL: https://issues.apache.org/jira/browse/HDDS-420 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Reporter: Nilotpal Nandi Fix For: 0.2.1 Here are the commands run : {noformat} [root@ctr-e138-1518143905142-468367-01-02 bin]# ./ozone oz -putKey /fs-volume/fs-bucket/nn1 -file /etc/passwd 2018-09-09 15:39:31,131 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Create key failed, error:KEY_ALLOCATION_ERROR [root@ctr-e138-1518143905142-468367-01-02 bin]# [root@ctr-e138-1518143905142-468367-01-02 bin]# ./ozone fs -copyFromLocal /etc/passwd / 2018-09-09 15:40:16,879 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-09-09 15:40:23,632 [main] ERROR - Try to allocate more blocks for write failed, already allocated 0 blocks for this write. copyFromLocal: Message missing required fields: keyLocation [root@ctr-e138-1518143905142-468367-01-02 bin]# ./ozone oz -putKey /fs-volume/fs-bucket/nn2 -file /etc/passwd 2018-09-09 15:44:55,912 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Create key failed, error:KEY_ALLOCATION_ERROR{noformat} hadoop version : --- {noformat} [root@ctr-e138-1518143905142-468367-01-02 bin]# ./hadoop version Hadoop 3.2.0-SNAPSHOT Source code repository git://git.apache.org/hadoop.git -r bf8a1750e99cfbfa76021ce51b6514c74c06f498 Compiled by root on 2018-09-08T10:22Z Compiled with protoc 2.5.0 >From source with checksum c5bbb375aed8edabd89c377af83189d This command was run using /root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT.jar{noformat} scm log : --- {noformat} 2018-09-09 15:45:00,907 INFO org.apache.hadoop.hdds.scm.pipelines.ratis.RatisManagerImpl: Allocating a new ratis pipeline of size: 3 id: pipelineId=f210716d-ba7b-4adf-91d6-da286e5fd010 2018-09-09 15:45:00,973 INFO org.apache.ratis.conf.ConfUtils: raft.rpc.type = GRPC (default) 2018-09-09 15:45:01,007 INFO org.apache.ratis.conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-09-09 15:45:01,011 INFO org.apache.ratis.conf.ConfUtils: raft.client.rpc.retryInterval = 300 ms (default) 2018-09-09 15:45:01,012 INFO org.apache.ratis.conf.ConfUtils: raft.client.async.outstanding-requests.max = 100 (default) 2018-09-09 15:45:01,012 INFO org.apache.ratis.conf.ConfUtils: raft.client.async.scheduler-threads = 3 (default) 2018-09-09 15:45:01,020 INFO org.apache.ratis.conf.ConfUtils: raft.grpc.flow.control.window = 1MB (=1048576) (default) 2018-09-09 15:45:01,020 INFO org.apache.ratis.conf.ConfUtils: raft.grpc.message.size.max = 33554432 (custom) 2018-09-09 15:45:01,102 INFO org.apache.ratis.conf.ConfUtils: raft.client.rpc.request.timeout = 3000 ms (default) 2018-09-09 15:45:01,667 ERROR org.apache.hadoop.hdds.scm.XceiverClientRatis: Failed to reinitialize RaftPeer:bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9:172.27.12.96:9858 datanode: bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9{ip: 172.27.12.96, host: ctr-e138-1518143905142-468367-01-07.hwx.site} org.apache.ratis.protocol.GroupMismatchException: bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9: The group (group-7347726F7570) of client-409D68EB500F does not match the group (group-2041ABBEE452) of the server bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.ratis.util.ReflectionUtils.instantiateException(ReflectionUtils.java:222) at org.apache.ratis.grpc.RaftGrpcUtil.tryUnwrapException(RaftGrpcUtil.java:79) at org.apache.ratis.grpc.RaftGrpcUtil.unwrapException(RaftGrpcUtil.java:67) at org.apache.ratis.grpc.client.RaftClientProtocolClient.blockingCall(RaftClientProtocolClient.java:127) at org.apache.ratis.grpc.client.RaftClientProtocolClient.reinitialize(RaftClientProtocolClient.java:102) at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:77) at org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:302) at org.apache.ratis.client.impl.RaftClientImpl.reinitialize(RaftClientImpl.java:216) at org.apache.hadoop.hdds.scm.XceiverClientRatis.reinitialize(XceiverClientRatis.java:163) at org.apache.hadoop.hdds.scm.XceiverClientRatis.reinitialize(XceiverClientRatis.java:133) a
[jira] [Created] (HDDS-451) PutKey failed due to error "Rejecting write chunk request. Chunk overwrite without explicit request"
Nilotpal Nandi created HDDS-451: --- Summary: PutKey failed due to error "Rejecting write chunk request. Chunk overwrite without explicit request" Key: HDDS-451 URL: https://issues.apache.org/jira/browse/HDDS-451 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Affects Versions: 0.2.1 Reporter: Nilotpal Nandi steps taken : -- # Ran Put Key command to write 50GB data. Put Key client operation failed after 17 mins. error seen ozone.log : {noformat} 2018-09-13 12:11:53,734 [ForkJoinPool.commonPool-worker-20] DEBUG (ChunkManagerImpl.java:85) - writing chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_1 chunk stage:COMMIT_DATA chunk file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_1 tmp chunk file 2018-09-13 12:11:56,576 [pool-3-thread-60] DEBUG (ChunkManagerImpl.java:85) - writing chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2 chunk stage:WRITE_DATA chunk file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2 tmp chunk file 2018-09-13 12:11:56,739 [ForkJoinPool.commonPool-worker-20] DEBUG (ChunkManagerImpl.java:85) - writing chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2 chunk stage:COMMIT_DATA chunk file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2 tmp chunk file 2018-09-13 12:12:21,410 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:148) - Executing cycle Number : 206 2018-09-13 12:12:51,411 [Datanode State Machine Thread - 0] DEBUG (DatanodeStateMachine.java:148) - Executing cycle Number : 207 2018-09-13 12:12:53,525 [BlockDeletingService#1] DEBUG (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next container, there is no pending deletion block contained in remaining containers. 2018-09-13 12:12:55,048 [Datanode ReportManager Thread - 1] DEBUG (ContainerSet.java:191) - Starting container report iteration. 2018-09-13 12:13:02,626 [pool-3-thread-1] ERROR (ChunkUtils.java:244) - Rejecting write chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2, offset=0, len=16777216} 2018-09-13 12:13:03,035 [pool-3-thread-1] INFO (ContainerUtils.java:149) - Operation: WriteChunk : Trace ID: 54834b29-603d-4ba9-9d68-0885215759d8 : Message: Rejecting write chunk request. OverWrite flag required.ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2, offset=0, len=16777216} : Result: OVERWRITE_FLAG_REQUIRED 2018-09-13 12:13:03,037 [ForkJoinPool.commonPool-worker-11] ERROR (ChunkUtils.java:244) - Rejecting write chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2, offset=0, len=16777216} 2018-09-13 12:13:03,037 [ForkJoinPool.commonPool-worker-11] INFO (ContainerUtils.java:149) - Operation: WriteChunk : Trace ID: 54834b29-603d-4ba9-9d68-0885215759d8 : Message: Rejecting write chunk request. OverWrite flag required.ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2, offset=0, len=16777216} : Result: OVERWRITE_FLAG_REQUIRED {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-458) numberofKeys is 0 for all containers even when keys are present
Nilotpal Nandi created HDDS-458: --- Summary: numberofKeys is 0 for all containers even when keys are present Key: HDDS-458 URL: https://issues.apache.org/jira/browse/HDDS-458 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Client Affects Versions: 0.2.1 Reporter: Nilotpal Nandi numberofKeys field is 0 for all containers even when keys are present {noformat} [root@ctr-e138-1518143905142-459606-01-05 bin]# ./ozone scmcli list --count=40 --start=1 | grep numberOfKeys "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0, "numberOfKeys" : 0,{noformat} {noformat} [root@ctr-e138-1518143905142-459606-01-05 bin]# ./ozone oz key list /fs-volume/fs-bucket/ | grep keyName 2018-09-13 19:10:33,502 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable "keyName" : "15GBFILE" "keyName" : "15GBFILE1" "keyName" : "1GB1" "keyName" : "1GB10" "keyName" : "1GB11" "keyName" : "1GB12" "keyName" : "1GB13" "keyName" : "1GB14" "keyName" : "1GB15" "keyName" : "1GB2" "keyName" : "1GB3" "keyName" : "1GB4" "keyName" : "1GB5" "keyName" : "1GB6" "keyName" : "1GB7" "keyName" : "1GB8" "keyName" : "1GB9" "keyName" : "1GBsecond1" "keyName" : "1GBsecond10" "keyName" : "1GBsecond11" "keyName" : "1GBsecond12" "keyName" : "1GBsecond13" "keyName" : "1GBsecond14" "keyName" : "1GBsecond15" "keyName" : "1GBsecond2" "keyName" : "1GBsecond3" "keyName" : "1GBsecond4" "keyName" : "1GBsecond5" "keyName" : "1GBsecond6" "keyName" : "1GBsecond7" "keyName" : "1GBsecond8" "keyName" : "1GBsecond9" "keyName" : "2GBFILE" "keyName" : "2GBFILE2" "keyName" : "50GBFILE2" "keyName" : "passwd1"{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-460) Replication manager failed to import container data
Nilotpal Nandi created HDDS-460: --- Summary: Replication manager failed to import container data Key: HDDS-460 URL: https://issues.apache.org/jira/browse/HDDS-460 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Affects Versions: 0.2.1 Reporter: Nilotpal Nandi replication manager is not able to import downloaded container data on datanode failure. snippet of ozone.log - {noformat} 2018-09-14 09:34:05,249 [grpc-default-executor-139] INFO (GrpcReplicationClient.java:161) - Container is downloaded to /tmp/container-copy/container-14.tar.gz 2018-09-14 09:34:05,389 [grpc-default-executor-131] ERROR (ReplicateContainerCommandHandler.java:164) - Can't import the downloaded container data id=8 java.io.EOFException at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.read(GzipCompressorInputStream.java:241) at org.apache.commons.compress.archivers.tar.TarBuffer.readBlock(TarBuffer.java:224) at org.apache.commons.compress.archivers.tar.TarBuffer.readRecord(TarBuffer.java:195) at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:486) at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.skip(TarArchiveInputStream.java:182) at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:220) at org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.unpackContainerDescriptor(TarContainerPacker.java:200) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.importContainer(ReplicateContainerCommandHandler.java:144) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.ReplicateContainerCommandHandler.lambda$handle$0(ReplicateContainerCommandHandler.java:121) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:656) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:632) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.ozone.container.replication.GrpcReplicationClient$StreamDownloader.onCompleted(GrpcReplicationClient.java:160) at org.apache.ratis.shaded.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418) at org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at org.apache.ratis.shaded.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) at org.apache.ratis.shaded.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at org.apache.ratis.shaded.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at org.apache.ratis.shaded.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403) at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) at org.apache.ratis.shaded.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584) at org.apache.ratis.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.ratis.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745){noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-461) container remains in CLOSING state forever
Nilotpal Nandi created HDDS-461: --- Summary: container remains in CLOSING state forever Key: HDDS-461 URL: https://issues.apache.org/jira/browse/HDDS-461 Project: Hadoop Distributed Data Store Issue Type: Bug Components: SCM Affects Versions: 0.2.1 Reporter: Nilotpal Nandi Container id # 13's state is not changing from CLOSING to CLOSED. {noformat} [root@ctr-e138-1518143905142-459606-01-02 bin]# ./ozone scmcli info 13 raft.rpc.type = GRPC (default) raft.grpc.message.size.max = 33554432 (custom) raft.client.rpc.retryInterval = 300 ms (default) raft.client.async.outstanding-requests.max = 100 (default) raft.client.async.scheduler-threads = 3 (default) raft.grpc.flow.control.window = 1MB (=1048576) (default) raft.grpc.message.size.max = 33554432 (custom) raft.client.rpc.request.timeout = 3000 ms (default) Container id: 13 Container State: OPEN Container Path: /tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/13/metadata Container Metadata: LeaderID: ctr-e138-1518143905142-459606-01-03.hwx.site Datanodes: [ctr-e138-1518143905142-459606-01-07.hwx.site,ctr-e138-1518143905142-459606-01-08.hwx.site,ctr-e138-1518143905142-459606-01-03.hwx.site]{noformat} snippet of scmcli list : {noformat} { "state" : "CLOSING", "replicationFactor" : "THREE", "replicationType" : "RATIS", "allocatedBytes" : 4831838208, "usedBytes" : 4831838208, "numberOfKeys" : 0, "lastUsed" : 4391827471, "stateEnterTime" : 5435591457, "owner" : "f8332db1-b8b1-4077-a9ea-097033d074b7", "containerID" : 13, "deleteTransactionId" : 0, "containerOpen" : true }{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-478) ozone Manager startup message is showing in ozone.log
Nilotpal Nandi created HDDS-478: --- Summary: ozone Manager startup message is showing in ozone.log Key: HDDS-478 URL: https://issues.apache.org/jira/browse/HDDS-478 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Affects Versions: 0.2.1 Reporter: Nilotpal Nandi Please find the snippet below. Ozone Manager startup message is showing up in ozone.log : {noformat} [root@ctr-e138-1518143905142-468367-01-02 logs]# cat ozone.log.2018-09-16 | head -20 2018-09-16 05:29:59,638 [main] INFO (LogAdapter.java:51) - STARTUP_MSG: / STARTUP_MSG: Starting OzoneManager STARTUP_MSG: host = ctr-e138-1518143905142-468367-01-02.hwx.site/172.27.68.129 STARTUP_MSG: args = [-createObjectStore] STARTUP_MSG: version = 3.2.0-SNAPSHOT STARTUP_MSG: classpath = /root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/etc/hadoop:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jersey-json-1.19.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jetty-webapp-9.3.19.v20170502.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/netty-3.10.5.Final.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jersey-servlet-1.19.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/token-provider-1.0.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/kerby-util-1.0.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/commons-lang3-3.7.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/curator-recipes-2.12.0.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/commons-math3-3.1.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/xz-1.0.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jackson-databind-2.9.5.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/guava-11.0.2.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/commons-cli-1.2.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/hadoop-annotations-3.2.0-SNAPSHOT.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/commons-collections-3.2.2.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/json-smart-2.3.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/commons-codec-1.11.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/curator-framework-2.12.0.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/gson-2.2.4.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jetty-server-9.3.19.v20170502.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jetty-security-9.3.19.v20170502.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jsp-api-2.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/log4j-1.2.17.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jersey-core-1.19.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/curator-client-2.12.0.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/stax2-api-3.1.4.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jetty-servlet-9.3.19.v20170502.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/kerb-common-1.0.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/kerb-core-1.0.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/commons-net-3.6.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/commons-logging-1.1.3.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jetty-http-9.3.19.v20170502.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/root/hadoop_trunk/ozone-0.3.0-SNAPSHOT/share/hadoop/common/lib/jsch-0.1.54.
[jira] [Created] (HDDS-479) Add more ozone fs tests in the robot integration framework
Nilotpal Nandi created HDDS-479: --- Summary: Add more ozone fs tests in the robot integration framework Key: HDDS-479 URL: https://issues.apache.org/jira/browse/HDDS-479 Project: Hadoop Distributed Data Store Issue Type: Test Reporter: Nilotpal Nandi Assignee: Nilotpal Nandi Fix For: 0.2.1 Currently , we have few number of ozone fs tests in robot integration framework. Need to add more. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-482) NullPointer exception thrown on console when cli operation failed
Nilotpal Nandi created HDDS-482: --- Summary: NullPointer exception thrown on console when cli operation failed Key: HDDS-482 URL: https://issues.apache.org/jira/browse/HDDS-482 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone CLI Affects Versions: 0.2.1 Reporter: Nilotpal Nandi Assignee: Nanda kumar steps taken : -- # running putKey from cli. # closed all containers. # Error thrown on cli - {noformat} Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.hdds.cli.GenericCli.printError(GenericCli.java:68) at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:54) at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:77) {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-493) ozone fs rmdir not working
Nilotpal Nandi created HDDS-493: --- Summary: ozone fs rmdir not working Key: HDDS-493 URL: https://issues.apache.org/jira/browse/HDDS-493 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.2.1 Reporter: Nilotpal Nandi ozone fs rmdir not working, also it does not throw any error on console. {noformat} [root@ctr-e138-1518143905142-468367-01-02 ~]# ozone fs -ls /TESTDIR3 2018-09-18 07:27:36,275 [main] DEBUG - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2018-09-18 07:27:37,103 [main] DEBUG - Call: public abstract org.apache.hadoop.ozone.client.OzoneVolume org.apache.hadoop.ozone.client.protocol.ClientProtocol.getVolumeDetails(java.lang.String) throws java.io.IOException took 29 ms 2018-09-18 07:27:37,300 [shutdown-hook-0] DEBUG - Call: public abstract void org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws java.io.IOException took 2 ms [root@ctr-e138-1518143905142-468367-01-02 ~]# [root@ctr-e138-1518143905142-468367-01-02 ~]# [root@ctr-e138-1518143905142-468367-01-02 ~]# ozone fs -rmdir /TESTDIR3 2018-09-18 07:27:51,320 [main] DEBUG - Using org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. 2018-09-18 07:27:52,108 [main] DEBUG - Call: public abstract org.apache.hadoop.ozone.client.OzoneVolume org.apache.hadoop.ozone.client.protocol.ClientProtocol.getVolumeDetails(java.lang.String) throws java.io.IOException took 22 ms 2018-09-18 07:27:52,312 [shutdown-hook-0] DEBUG - Call: public abstract void org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws java.io.IOException took 1 ms {noformat} ozone.log - {noformat} 2018-09-18 07:27:52,298 [IPC Server handler 2 on 9889] DEBUG (KeyManagerImpl.java:342) - volume:fs-volume bucket:fs-bucket Key:TESTDIR3 not found 2018-09-18 07:27:52,298 [IPC Server handler 2 on 9889] ERROR (KeyManagerImpl.java:349) - Get key failed for volume:fs-volume bucket:fs-bucket key:TESTDIR3 org.apache.hadoop.ozone.om.exceptions.OMException: Key not found at org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:344) at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:853) at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:384) at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:35895) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) 2018-09-18 07:27:52,305 [IPC Server handler 4 on 9889] DEBUG (KeyManagerImpl.java:452) - Key TESTDIR3/ deleted from OM DB 2018-09-18 07:27:52,312 [shutdown-hook-0] DEBUG (OzoneClientInvocationHandler.java:55) - Call: public abstract void org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws java.io.IOException took 1 ms{noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-492) Add more unit tests to ozonefs robot framework
[ https://issues.apache.org/jira/browse/HDDS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nilotpal Nandi resolved HDDS-492. - Resolution: Duplicate > Add more unit tests to ozonefs robot framework > -- > > Key: HDDS-492 > URL: https://issues.apache.org/jira/browse/HDDS-492 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Namit Maheshwari >Assignee: Namit Maheshwari >Priority: Major > > Currently there are only couple of tests inside ozonefs.robot > We should add more unit tests for the same. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-493) ozone fs rmdir not working
[ https://issues.apache.org/jira/browse/HDDS-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nilotpal Nandi resolved HDDS-493. - Resolution: Not A Problem > ozone fs rmdir not working > -- > > Key: HDDS-493 > URL: https://issues.apache.org/jira/browse/HDDS-493 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.2.1 >Reporter: Nilotpal Nandi >Assignee: Nilotpal Nandi >Priority: Major > > > ozone fs rmdir not working, also it does not throw any error on console. > {noformat} > > > [root@ctr-e138-1518143905142-468367-01-02 ~]# ozone fs -ls /TESTDIR3 > 2018-09-18 07:27:36,275 [main] DEBUG - Using > org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. > 2018-09-18 07:27:37,103 [main] DEBUG - Call: public abstract > org.apache.hadoop.ozone.client.OzoneVolume > org.apache.hadoop.ozone.client.protocol.ClientProtocol.getVolumeDetails(java.lang.String) > throws java.io.IOException took 29 ms > 2018-09-18 07:27:37,300 [shutdown-hook-0] DEBUG - Call: public abstract void > org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws > java.io.IOException took 2 ms > [root@ctr-e138-1518143905142-468367-01-02 ~]# > [root@ctr-e138-1518143905142-468367-01-02 ~]# > [root@ctr-e138-1518143905142-468367-01-02 ~]# ozone fs -rmdir /TESTDIR3 > 2018-09-18 07:27:51,320 [main] DEBUG - Using > org.apache.hadoop.ozone.client.rpc.RpcClient as client protocol. > 2018-09-18 07:27:52,108 [main] DEBUG - Call: public abstract > org.apache.hadoop.ozone.client.OzoneVolume > org.apache.hadoop.ozone.client.protocol.ClientProtocol.getVolumeDetails(java.lang.String) > throws java.io.IOException took 22 ms > 2018-09-18 07:27:52,312 [shutdown-hook-0] DEBUG - Call: public abstract void > org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws > java.io.IOException took 1 ms > {noformat} > > > ozone.log > - > {noformat} > 2018-09-18 07:27:52,298 [IPC Server handler 2 on 9889] DEBUG > (KeyManagerImpl.java:342) - volume:fs-volume bucket:fs-bucket Key:TESTDIR3 > not found > 2018-09-18 07:27:52,298 [IPC Server handler 2 on 9889] ERROR > (KeyManagerImpl.java:349) - Get key failed for volume:fs-volume > bucket:fs-bucket key:TESTDIR3 > org.apache.hadoop.ozone.om.exceptions.OMException: Key not found > at > org.apache.hadoop.ozone.om.KeyManagerImpl.lookupKey(KeyManagerImpl.java:344) > at org.apache.hadoop.ozone.om.OzoneManager.lookupKey(OzoneManager.java:853) > at > org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.lookupKey(OzoneManagerProtocolServerSideTranslatorPB.java:384) > at > org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java:35895) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682) > 2018-09-18 07:27:52,305 [IPC Server handler 4 on 9889] DEBUG > (KeyManagerImpl.java:452) - Key TESTDIR3/ deleted from OM DB > 2018-09-18 07:27:52,312 [shutdown-hook-0] DEBUG > (OzoneClientInvocationHandler.java:55) - Call: public abstract void > org.apache.hadoop.ozone.client.protocol.ClientProtocol.close() throws > java.io.IOException took 1 ms{noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-494) client fails to write when all the containers are closed through scmcli
Nilotpal Nandi created HDDS-494: --- Summary: client fails to write when all the containers are closed through scmcli Key: HDDS-494 URL: https://issues.apache.org/jira/browse/HDDS-494 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.2.1 Reporter: Nilotpal Nandi steps taken : -- # running putKey from cli. # closed all containers. # Client fails to write and throws error on cli (related to bug HDDS-482 ) Expectation : --- there should be retry and allocate new containers . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-543) datanode service start throwing NullPointerException
Nilotpal Nandi created HDDS-543: --- Summary: datanode service start throwing NullPointerException Key: HDDS-543 URL: https://issues.apache.org/jira/browse/HDDS-543 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi steps taken : --- # Stopped one of the nodes in 6 node cluster while write operation was running. # Tried to start datanode after sometime. Datanode failed to start ozone version : - {noformat} Source code repository g...@github.com:apache/hadoop.git -r 968082ffa5d9e50ed8538f653c375edd1b8feea5 Compiled by elek on 2018-09-19T20:57Z Compiled with protoc 2.5.0 >From source with checksum efbdeabb5670d69d9efde85846e4ee98 Using HDDS 0.2.1-alpha Source code repository g...@github.com:apache/hadoop.git -r 968082ffa5d9e50ed8538f653c375edd1b8feea5 Compiled by elek on 2018-09-19T20:56Z Compiled with protoc 2.5.0 >From source with checksum 8bf78cff4b73c95d486da5b21053ef {noformat} console log : {noformat} 2018-09-24 09:53:54,913 INFO impl.RaftServerImpl: 9348f4cb-0bd3-41e7-b1c6-c22463a20c09: set configuration 0: [0ba4abe0-c8d1-4d6c-8d28-ebfa08fb84c9:172.27.18.18:9858, 3d6c8b2c-f1d2-4425-a577-b9ef5dc7c634:172.27.80.23:9858, 9348f4cb-0bd3-41e7-b1c6-c22463a20c09:172.27.23.161:9858], old=null at 0 Sep 24, 2018 9:53:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM' 2018-09-24 09:53:55,968 [main] INFO - Listening HDDS REST traffic on /0.0.0.0:9880 2018-09-24 09:53:55,969 [main] INFO - Started plug-in org.apache.hadoop.ozone.web.OzoneHddsDatanodeService@2fa3be26 2018-09-24 09:53:55,975 [Datanode State Machine Thread - 0] INFO - Attempting to start container services. 2018-09-24 09:53:56,171 [Datanode State Machine Thread - 0] INFO - Starting XceiverServerRatis 9348f4cb-0bd3-41e7-b1c6-c22463a20c09 at port 9858 2018-09-24 09:53:56,193 [Datanode State Machine Thread - 0] ERROR - Unable to start the DatanodeState Machine java.io.IOException: Premature EOF from inputStream at org.apache.ratis.util.IOUtils.readFully(IOUtils.java:100) at org.apache.ratis.server.storage.LogReader.decodeEntry(LogReader.java:250) at org.apache.ratis.server.storage.LogReader.readEntry(LogReader.java:155) at org.apache.ratis.server.storage.LogInputStream.nextEntry(LogInputStream.java:128) at org.apache.ratis.server.storage.LogSegment.readSegmentFile(LogSegment.java:110) at org.apache.ratis.server.storage.LogSegment.loadSegment(LogSegment.java:132) at org.apache.ratis.server.storage.RaftLogCache.loadSegment(RaftLogCache.java:110) at org.apache.ratis.server.storage.SegmentedRaftLog.loadLogSegments(SegmentedRaftLog.java:155) at org.apache.ratis.server.storage.SegmentedRaftLog.open(SegmentedRaftLog.java:123) at org.apache.ratis.server.impl.ServerState.initLog(ServerState.java:162) at org.apache.ratis.server.impl.ServerState.(ServerState.java:110) at org.apache.ratis.server.impl.RaftServerImpl.(RaftServerImpl.java:106) at org.apache.ratis.server.impl.RaftServerProxy.lambda$newRaftServerImpl$0(RaftServerProxy.java:191) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) 2018-09-24 09:53:56,195 [main] ERROR - Exception in HddsDatanodeService. java.lang.NullPointerException at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.join(DatanodeStateMachine.java:332) at org.apache.hadoop.ozone.HddsDatanodeService.join(HddsDatanodeService.java:191) at org.apache.hadoop.ozone.HddsDatanodeService.main(HddsDatanodeService.java:250) 2018-09-24 09:53:56,198 INFO util.ExitUtil: Exiting with status 1: java.lang.NullPointerException 2018-09-24 09:53:56,201 WARN fs.CachingGetSpaceUsed: Thread Interrupted waiting to refresh disk information: sleep interrupted 2018-09-24 09:53:56,201 [shutdown-hook-0] INFO - SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down HddsDatanodeService at ctr-e138-1518143905142-468367-01-03.hwx.site/172.27.23.161 {noformat} datanode.log -- {noformat} 2018-09-24 09:42:41,305 ERROR org.apache.ratis.server.impl.RaftServerImpl: 9348f4cb-0bd3-41e7-b1c6-c22463a20c09: applyTransaction failed for index:2 proto:(t:2, i:2)SMLOGENTRY, client-DBCE827E62E6, cid=1 2018-09-24 09:42:41,307 ERROR org.apache.ratis.server.impl.StateMachineUpdater: Terminating with exit
[jira] [Created] (HDDS-545) NullPointerException error thrown while trying to close container
Nilotpal Nandi created HDDS-545: --- Summary: NullPointerException error thrown while trying to close container Key: HDDS-545 URL: https://issues.apache.org/jira/browse/HDDS-545 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Nilotpal Nandi Seen following null pointer in ozone.log while trying to close the container on receiving SCM container close request'. ozone version: -- {noformat} Source code repository g...@github.com:apache/hadoop.git -r 968082ffa5d9e50ed8538f653c375edd1b8feea5 Compiled by elek on 2018-09-19T20:57Z Compiled with protoc 2.5.0 >From source with checksum efbdeabb5670d69d9efde85846e4ee98 Using HDDS 0.2.1-alpha Source code repository g...@github.com:apache/hadoop.git -r 968082ffa5d9e50ed8538f653c375edd1b8feea5 Compiled by elek on 2018-09-19T20:56Z Compiled with protoc 2.5.0 >From source with checksum 8bf78cff4b73c95d486da5b21053ef {noformat} ozone.log {noformat} 2018-09-24 11:32:55,910 [Thread-2921] DEBUG (XceiverServerRatis.java:401) - pipeline Action CLOSE on pipeline pipelineId=eabdcbe2-da3b-41be-a281-f0ea8d4120f7.Reason : 7d1c7be2-7882-4446-be61-be868d2e188a is in candidate state for 1074164ms 2018-09-24 11:32:56,343 [Datanode State Machine Thread - 1] DEBUG (HeartbeatEndpointTask.java:255) - Received SCM container close request for container 54 2018-09-24 11:32:56,347 [Datanode State Machine Thread - 1] DEBUG (HeartbeatEndpointTask.java:255) - Received SCM container close request for container 42 2018-09-24 11:32:56,347 [Datanode State Machine Thread - 1] DEBUG (HeartbeatEndpointTask.java:255) - Received SCM container close request for container 44 2018-09-24 11:32:56,354 [Datanode State Machine Thread - 1] DEBUG (HeartbeatEndpointTask.java:255) - Received SCM container close request for container 46 2018-09-24 11:32:56,355 [Datanode State Machine Thread - 1] DEBUG (HeartbeatEndpointTask.java:255) - Received SCM container close request for container 48 2018-09-24 11:32:56,357 [Datanode State Machine Thread - 1] DEBUG (HeartbeatEndpointTask.java:255) - Received SCM container close request for container 50 2018-09-24 11:32:56,357 [Datanode State Machine Thread - 1] DEBUG (HeartbeatEndpointTask.java:255) - Received SCM container close request for container 52 2018-09-24 11:32:56,548 [Command processor thread] DEBUG (CloseContainerCommandHandler.java:64) - Processing Close Container command. 2018-09-24 11:32:56,636 [Command processor thread] ERROR (CloseContainerCommandHandler.java:105) - Can't close container 54 java.lang.NullPointerException at org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.submitContainerRequest(OzoneContainer.java:192) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:91) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:93) at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$1(DatanodeStateMachine.java:382) at java.lang.Thread.run(Thread.java:745) 2018-09-24 11:32:56,726 [Command processor thread] DEBUG (CloseContainerCommandHandler.java:64) - Processing Close Container command. 2018-09-24 11:32:56,728 [Command processor thread] ERROR (CloseContainerCommandHandler.java:105) - Can't close container 42 java.lang.NullPointerException at org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.submitContainerRequest(OzoneContainer.java:192) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:91) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:93) at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$1(DatanodeStateMachine.java:382) at java.lang.Thread.run(Thread.java:745) 2018-09-24 11:32:56,787 [Command processor thread] DEBUG (CloseContainerCommandHandler.java:64) - Processing Close Container command. 2018-09-24 11:32:56,814 [Command processor thread] ERROR (CloseContainerCommandHandler.java:105) - Can't close container 44 java.lang.NullPointerException at org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.submitContainerRequest(OzoneContainer.java:192) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CloseContainerCommandHandler.handle(CloseContainerCommandHandler.java:91) at org.apache.hadoop.ozone.container.common.statemachine.commandhandler.CommandDispatcher.handle(CommandDispatcher.java:93) at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$initCommandHandlerThread$1(DatanodeStateMachine.java:382) at java.lang.Thread.run(Thread.java:745) 2
[jira] [Created] (HDDS-553) copyFromLocal subcommand failed
Nilotpal Nandi created HDDS-553: --- Summary: copyFromLocal subcommand failed Key: HDDS-553 URL: https://issues.apache.org/jira/browse/HDDS-553 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.3.0 Reporter: Nilotpal Nandi copyFromLocal subcommand is throwing error when run from docker container : {noformat} hadoop@bf5c576c7556:~$ ozone fs -copyFromLocal /etc/passwd o3://bucket1.fstest/testdir/deep/ 2018-09-25 18:59:24 ERROR ChunkGroupOutputStream:274 - Try to allocate more blocks for write failed, already allocated 0 blocks for this write. copyFromLocal: Allocate block failed, error:INTERNAL_ERROR {noformat} ozone version {noformat} Source code repository git://git.apache.org/hadoop.git -r 26c94a0fd07dab40fb5e62614ae71d98110a7467 Compiled by nnandi on 2018-09-25T18:30Z Compiled with protoc 2.5.0 >From source with checksum 1703a7974737c3160f49de7ef4f435b Using HDDS 0.3.0-SNAPSHOT Source code repository git://git.apache.org/hadoop.git -r 26c94a0fd07dab40fb5e62614ae71d98110a7467 Compiled by nnandi on 2018-09-25T18:29Z Compiled with protoc 2.5.0 >From source with checksum 1785735bba1697df364730122de197b {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-553) copyFromLocal subcommand failed
[ https://issues.apache.org/jira/browse/HDDS-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nilotpal Nandi resolved HDDS-553. - Resolution: Not A Problem > copyFromLocal subcommand failed > --- > > Key: HDDS-553 > URL: https://issues.apache.org/jira/browse/HDDS-553 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Nilotpal Nandi >Assignee: Shashikant Banerjee >Priority: Major > > copyFromLocal subcommand is throwing error when run from docker container : > > {noformat} > hadoop@bf5c576c7556:~$ ozone fs -copyFromLocal /etc/passwd > o3://bucket1.fstest/testdir/deep/ > 2018-09-25 18:59:24 ERROR ChunkGroupOutputStream:274 - Try to allocate more > blocks for write failed, already allocated 0 blocks for this write. > copyFromLocal: Allocate block failed, error:INTERNAL_ERROR > {noformat} > > ozone version > > {noformat} > Source code repository git://git.apache.org/hadoop.git -r > 26c94a0fd07dab40fb5e62614ae71d98110a7467 > Compiled by nnandi on 2018-09-25T18:30Z > Compiled with protoc 2.5.0 > From source with checksum 1703a7974737c3160f49de7ef4f435b > Using HDDS 0.3.0-SNAPSHOT > Source code repository git://git.apache.org/hadoop.git -r > 26c94a0fd07dab40fb5e62614ae71d98110a7467 > Compiled by nnandi on 2018-09-25T18:29Z > Compiled with protoc 2.5.0 > From source with checksum 1785735bba1697df364730122de197b > {noformat} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org