[jira] [Created] (HDFS-15602) Support new Instance by non default constructor by ReflectionUtils
maobaolong created HDFS-15602: - Summary: Support new Instance by non default constructor by ReflectionUtils Key: HDFS-15602 URL: https://issues.apache.org/jira/browse/HDFS-15602 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.4.0 Reporter: maobaolong Assignee: maobaolong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15399) Support include or exclude datanode by configure file
maobaolong created HDFS-15399: - Summary: Support include or exclude datanode by configure file Key: HDFS-15399 URL: https://issues.apache.org/jira/browse/HDFS-15399 Project: Hadoop HDFS Issue Type: New Feature Components: datanode Reporter: maobaolong Assignee: maobaolong When i dislike a datanode, or just want to let specific datanode join to SCM, i want to have this feature to limit datanode list. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15139) Use RDBStore and TypedTable to manage the blockinfo of namenode
maobaolong created HDFS-15139: - Summary: Use RDBStore and TypedTable to manage the blockinfo of namenode Key: HDFS-15139 URL: https://issues.apache.org/jira/browse/HDFS-15139 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.3.0 Reporter: maobaolong replace the BlockManager.BlocksMap.blocks from GSet to rocksdb -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15138) Use RDBStore and TypedTable to manage the inode of namenode
maobaolong created HDFS-15138: - Summary: Use RDBStore and TypedTable to manage the inode of namenode Key: HDFS-15138 URL: https://issues.apache.org/jira/browse/HDFS-15138 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.3.0 Reporter: maobaolong Replace FSDirectory.inodeMap.map from GSet to rocksdb. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15137) Move RDBStore logic from apache-ozone into hadoop-commons module of apache-hadoop
maobaolong created HDFS-15137: - Summary: Move RDBStore logic from apache-ozone into hadoop-commons module of apache-hadoop Key: HDFS-15137 URL: https://issues.apache.org/jira/browse/HDFS-15137 Project: Hadoop HDFS Issue Type: Sub-task Reporter: maobaolong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-15133) Use rocksdb to store NameNode inode and blockInfo
maobaolong created HDFS-15133: - Summary: Use rocksdb to store NameNode inode and blockInfo Key: HDFS-15133 URL: https://issues.apache.org/jira/browse/HDFS-15133 Project: Hadoop HDFS Issue Type: Improvement Reporter: maobaolong Maybe we don't need checkpoint to a fsimage file, the rocksdb checkpoint can achieve the same request. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDDS-1606) ozone s3g cannot started caused by NoInitialContextException: xxx java.naming.factory.initial
maobaolong created HDDS-1606: Summary: ozone s3g cannot started caused by NoInitialContextException: xxx java.naming.factory.initial Key: HDDS-1606 URL: https://issues.apache.org/jira/browse/HDDS-1606 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.5.0 Environment: ozone-site.xml {code:xml} ozone.enabled true ozone.metadata.dirs /data0/disk1/meta ozone.scm.datanode.id /data0/disk1/meta/node/datanode.id ozone.om.address ozonemanager.hadoop.apache.org ozone.om.db.dirs /data0/om-db-dirs ozone.scm.names 172.16.150.142 ozone.om.address 172.16.150.142 hdds.datanode.http.enabled true ozone.s3g.domain.name s3g.internal {code} Reporter: maobaolong $ ozone s3g /software/servers/jdk1.8.0_121/bin/java -Dproc_s3g -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5008 -Dhadoop.log.dir=/software/servers/ozone-0.5.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/software/servers/ozone-0.5.0-SNAPSHOT -Dhadoop.id.str=hadp -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.ozone.s3.Gateway 2019-05-29 16:46:28,056 INFO hdfs.DFSUtil: Starting Web-server for s3gateway at: http://0.0.0.0:9878 2019-05-29 16:46:28,079 INFO util.log: Logging initialized @8123ms 2019-05-29 16:46:28,164 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2019-05-29 16:46:28,178 INFO http.HttpRequestLog: Http request log for http.requests.s3gateway is not defined 2019-05-29 16:46:28,188 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2019-05-29 16:46:28,191 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context s3gateway 2019-05-29 16:46:28,191 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2019-05-29 16:46:28,191 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2019-05-29 16:46:28,206 [main] INFO - Starting Ozone S3 gateway 2019-05-29 16:46:28,212 INFO http.HttpServer2: Jetty bound to port 9878 2019-05-29 16:46:28,213 INFO server.Server: jetty-9.3.24.v20180605, build timestamp: 2018-06-06T01:11:56+08:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827 2019-05-29 16:46:28,241 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@68f4865{/logs,file:///software/servers/ozone-0.5.0-SNAPSHOT/logs/,AVAILABLE} 2019-05-29 16:46:28,242 INFO handler.ContextHandler: Started o.e.j.s.ServletContextHandler@39d9314d{/static,jar:file:/software/servers/ozone-0.5.0-SNAPSHOT/share/ozone/lib/hadoop-ozone-s3gateway-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE} ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2 2019-05-29 16:46:28,974 WARN webapp.WebAppContext: Failed startup of context o.e.j.w.WebAppContext@7487b142{/,file:///tmp/jetty-0.0.0.0-9878-s3gateway-_-any-2799631504400193724.dir/webapp/,UNAVAILABLE}{/s3gateway} org.jboss.weld.exceptions.DefinitionException: Exception List with 1 exceptions: Exception 0 : java.lang.RuntimeException: javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.initial at com.sun.jersey.server.impl.cdi.CDIExtension.initialize(CDIExtension.java:201) at com.sun.jersey.server.impl.cdi.CDIExtension.beforeBeanDiscovery(CDIExtension.java:302) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) at org.jboss.weld.injection.MethodInvocationStrategy$SpecialParamPlusBeanManagerStrategy.invoke(MethodInvocationStrategy.java:144) at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:29
[jira] [Created] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
maobaolong created HDFS-14353: - Summary: Erasure Coding: metrics xmitsInProgress become to negative. Key: HDFS-14353 URL: https://issues.apache.org/jira/browse/HDFS-14353 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, erasure-coding Affects Versions: 3.3.0 Reporter: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14344) Erasure Coding: Miss EC block after decommission and restart NN
maobaolong created HDFS-14344: - Summary: Erasure Coding: Miss EC block after decommission and restart NN Key: HDFS-14344 URL: https://issues.apache.org/jira/browse/HDFS-14344 Project: Hadoop HDFS Issue Type: Sub-task Components: ec, erasure-coding, namenode Affects Versions: 3.3.0 Reporter: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13881) Export or Import a dirImage
maobaolong created HDFS-13881: - Summary: Export or Import a dirImage Key: HDFS-13881 URL: https://issues.apache.org/jira/browse/HDFS-13881 Project: Hadoop HDFS Issue Type: New Feature Components: namenode Affects Versions: 3.1.1 Reporter: maobaolong Assignee: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13804) DN maxDataLength is useless except DN webui. I suggest to get maxDataLength from NN heartbeat.
maobaolong created HDFS-13804: - Summary: DN maxDataLength is useless except DN webui. I suggest to get maxDataLength from NN heartbeat. Key: HDFS-13804 URL: https://issues.apache.org/jira/browse/HDFS-13804 Project: Hadoop HDFS Issue Type: New Feature Components: datanode Reporter: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13783) Balancer: make balancer to be a long service process for easy to monitor it.
maobaolong created HDFS-13783: - Summary: Balancer: make balancer to be a long service process for easy to monitor it. Key: HDFS-13783 URL: https://issues.apache.org/jira/browse/HDFS-13783 Project: Hadoop HDFS Issue Type: New Feature Components: balancer & mover Affects Versions: 3.0.3 Reporter: maobaolong If we have a long service process of balancer, like namenode, datanode, we can get metrics of balancer, the metrics can tell us the status of balancer, the amount of block it has moved, We can get or set the balance plan by the balancer webUI. So many things we can do if we have a long balancer service process. So, shall we start to plan the new Balancer? Hope this feature can enter the next release of hadoop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13527) craeteLocatedBlock IsCorrupt logic is fault when all block are corrupt.
maobaolong created HDFS-13527: - Summary: craeteLocatedBlock IsCorrupt logic is fault when all block are corrupt. Key: HDFS-13527 URL: https://issues.apache.org/jira/browse/HDFS-13527 Project: Hadoop HDFS Issue Type: Bug Components: hdfs, namenode Affects Versions: 3.2.0 Reporter: maobaolong the step is: 1. put a small file into hdfs FILEPATH 2. remove block replicas in all datanode blockpool. 3. restart datanode 4. restart namenode( leave safemode) 5. hdfs fsck FILEPATH -files -blocks -locations 6. namenode think this block is not corrupt block. the code logic is: {code:java} // get block locations NumberReplicas numReplicas = countNodes(blk); final int numCorruptNodes = numReplicas.corruptReplicas(); final int numCorruptReplicas = corruptReplicas.numCorruptReplicas(blk); if (numCorruptNodes != numCorruptReplicas) { LOG.warn("Inconsistent number of corrupt replicas for {}" + " blockMap has {} but corrupt replicas map has {}", blk, numCorruptNodes, numCorruptReplicas); } final int numNodes = blocksMap.numNodes(blk); final boolean isCorrupt; if (blk.isStriped()) { BlockInfoStriped sblk = (BlockInfoStriped) blk; isCorrupt = numCorruptReplicas != 0 && numReplicas.liveReplicas() < sblk.getRealDataBlockNum(); } else { isCorrupt = numCorruptReplicas != 0 && numCorruptReplicas == numNodes; } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13480) RBF: separate namenodeHeartbeat and routerHeartbeat to different config key.
maobaolong created HDFS-13480: - Summary: RBF: separate namenodeHeartbeat and routerHeartbeat to different config key. Key: HDFS-13480 URL: https://issues.apache.org/jira/browse/HDFS-13480 Project: Hadoop HDFS Issue Type: Bug Reporter: maobaolong Assignee: maobaolong Now, if i enable the heartbeat.enable, but i do not want to monitor any namenode, i get an ERROR log like: {code:java} [2018-04-19T14:00:03.057+08:00] [ERROR] federation.router.Router.serviceInit(Router.java 214) [main] : Heartbeat is enabled but there are no namenodes to monitor {code} and if i disable the heartbeat.enable, we cannot get any mounttable update, because the following logic in Router.java: {code:java} if (conf.getBoolean( RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE, RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE_DEFAULT)) { // Create status updater for each monitored Namenode this.namenodeHeartbeatServices = createNamenodeHeartbeatServices(); for (NamenodeHeartbeatService hearbeatService : this.namenodeHeartbeatServices) { addService(hearbeatService); } if (this.namenodeHeartbeatServices.isEmpty()) { LOG.error("Heartbeat is enabled but there are no namenodes to monitor"); } // Periodically update the router state this.routerHeartbeatService = new RouterHeartbeatService(this); addService(this.routerHeartbeatService); } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong resolved HDFS-13293. --- Resolution: Duplicate > RBF: The RouterRPCServer should transfer CallerContext and client ip to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Priority: Major > > Otherwise, the namenode don't know the client's callerContext -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13387) Make multi-thread access class thread safe
maobaolong created HDFS-13387: - Summary: Make multi-thread access class thread safe Key: HDFS-13387 URL: https://issues.apache.org/jira/browse/HDFS-13387 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.2.0 Reporter: maobaolong Assignee: maobaolong This jira will lead us to make the class as BlockInfoContiguous thread-safe, then, we should not use the NameSystemLock to lock the full flow. This just a base step to achieve the plan of HDFS-8966 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13293) RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer
maobaolong created HDFS-13293: - Summary: RBF: The RouterRPCServer should transfer CallerContext and client ip to NamenodeRpcServer Key: HDFS-13293 URL: https://issues.apache.org/jira/browse/HDFS-13293 Project: Hadoop HDFS Issue Type: Sub-task Reporter: maobaolong Otherwise, the namenode don't know the client's callerContext -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13278) Correct the logic of mount validate to avoid the bad mountPoint
maobaolong created HDFS-13278: - Summary: Correct the logic of mount validate to avoid the bad mountPoint Key: HDFS-13278 URL: https://issues.apache.org/jira/browse/HDFS-13278 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs Affects Versions: 3.2.0 Reporter: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13270) RBF: Router audit logger
maobaolong created HDFS-13270: - Summary: RBF: Router audit logger Key: HDFS-13270 URL: https://issues.apache.org/jira/browse/HDFS-13270 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs Affects Versions: 3.2.0 Reporter: maobaolong We can use router auditlogger to log the client info and cmd, because the FSNamesystem#Auditlogger's log think the client are all from router. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13269) After too many open file exception occurred, the standby NN never do checkpoint
maobaolong created HDFS-13269: - Summary: After too many open file exception occurred, the standby NN never do checkpoint Key: HDFS-13269 URL: https://issues.apache.org/jira/browse/HDFS-13269 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 3.2.0 Reporter: maobaolong do saveNameSpace in dfsadmin. The output as following: {code:java} saveNamespace: No image directories available! {code} The Namenode log show: {code:java} [2018-01-13T10:32:19.903+08:00] [INFO] [Standby State Checkpointer] : Triggering checkpoint because there have been 10159265 txns since the last checkpoint, which exceeds the configured threshold 1000 [2018-01-13T10:32:19.903+08:00] [INFO] [Standby State Checkpointer] : Save namespace ... ... [2018-01-13T10:37:10.539+08:00] [WARN] [1985938863@qtp-61073295-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@HOST_A:50070] : EXCEPTION java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:686) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) [2018-01-13T10:37:15.421+08:00] [ERROR] [FSImageSaver for /data0/nn of type IMAGE_AND_EDITS] : Unable to save image for /data0/nn java.io.FileNotFoundException: /data0/nn/current/fsimage_40247283317.md5.tmp (Too many open files) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at java.io.FileOutputStream.(FileOutputStream.java:162) at org.apache.hadoop.hdfs.util.AtomicFileOutputStream.(AtomicFileOutputStream.java:58) at org.apache.hadoop.hdfs.util.MD5FileUtils.saveMD5File(MD5FileUtils.java:157) at org.apache.hadoop.hdfs.util.MD5FileUtils.saveMD5File(MD5FileUtils.java:149) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:990) at org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:1039) at java.lang.Thread.run(Thread.java:745) [2018-01-13T10:37:15.421+08:00] [ERROR] [Standby State Checkpointer] : Error reported on storage directory Storage Directory /data0/nn [2018-01-13T10:37:15.421+08:00] [WARN] [Standby State Checkpointer] : About to remove corresponding storage: /data0/nn [2018-01-13T10:37:15.429+08:00] [ERROR] [Standby State Checkpointer] : Exception in doCheckpoint java.io.IOException: Failed to save in any storage directories while saving namespace. at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1176) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:1107) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(StandbyCheckpointer.java:185) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.access$1400(StandbyCheckpointer.java:62) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.doWork(StandbyCheckpointer.java:353) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.access$700(StandbyCheckpointer.java:260) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread$1.run(StandbyCheckpointer.java:280) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$CheckpointerThread.run(StandbyCheckpointer.java:276) ... [2018-01-13T15:52:33.783+08:00] [INFO] [Standby State Checkpointer] : Save namespace ... [2018-01-13T15:52:33.783+08:00] [ERROR] [Standby State Checkpointer] : Exception in doCheckpoint java.io.IOException: No image directories available! at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:1152) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:1107) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.doCheckpoint(StandbyCheckpointer.java:185) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer.access$
[jira] [Created] (HDFS-13245) RBF: State store DBMS implementation
maobaolong created HDFS-13245: - Summary: RBF: State store DBMS implementation Key: HDFS-13245 URL: https://issues.apache.org/jira/browse/HDFS-13245 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs Reporter: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 in use
maobaolong created HDFS-13241: - Summary: RBF: TestRouterSafemode failed if the port in use Key: HDFS-13241 URL: https://issues.apache.org/jira/browse/HDFS-13241 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs, test Affects Versions: 3.2.0 Reporter: maobaolong Assignee: maobaolong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
maobaolong created HDFS-13226: - Summary: RBF: We should throw the failure validate and refuse this mount entry Key: HDFS-13226 URL: https://issues.apache.org/jira/browse/HDFS-13226 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs Affects Versions: 3.2.0 Reporter: maobaolong one of the mount entry source path rule is that the source path must start with '\', somebody didn't follow the rule and execute the following command: {code:bash} $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/ {code} But, the console show we are successful add this entry. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13199) Fix the hdfs router page missing label icon issue
maobaolong created HDFS-13199: - Summary: Fix the hdfs router page missing label icon issue Key: HDFS-13199 URL: https://issues.apache.org/jira/browse/HDFS-13199 Project: Hadoop HDFS Issue Type: Bug Components: federation, hdfs Affects Versions: 3.0.0, 3.2.0 Reporter: maobaolong This bug is a typo error. decommisioned should be decommissioned -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
maobaolong created HDFS-13195: - Summary: DataNode conf page cannot display the current value after reconfig Key: HDFS-13195 URL: https://issues.apache.org/jira/browse/HDFS-13195 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: maobaolong Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i reconfig this key, the conf page's value is still the old config value. The reason is that: {code:java} public DatanodeHttpServer(final Configuration conf, final DataNode datanode, final ServerSocketChannel externalHttpChannel) throws IOException { this.conf = conf; Configuration confForInfoServer = new Configuration(conf); confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10); HttpServer2.Builder builder = new HttpServer2.Builder() .setName("datanode") .setConf(confForInfoServer) .setACL(new AccessControlList(conf.get(DFS_ADMIN, " "))) .hostName(getHostnameForSpnegoPrincipal(confForInfoServer)) .addEndpoint(URI.create("http://localhost:0";)) .setFindPort(true); this.infoServer = builder.build(); {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13034) The DFSUsed value bigger than the Capacity
[ https://issues.apache.org/jira/browse/HDFS-13034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong resolved HDFS-13034. --- Resolution: Won't Fix This situation may be amazing, but it is not a issue. > The DFSUsed value bigger than the Capacity > -- > > Key: HDFS-13034 > URL: https://issues.apache.org/jira/browse/HDFS-13034 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: maobaolong >Priority: Minor > > ||Node||Last contact||Admin State||Capacity||Used||Non DFS > Used||Remaining||Blocks||Block pool used||Failed Volumes||Version|| > |A|0|In Service|20.65 TB|18.26 TB|0 B|1.27 TB|24330|2.57 TB (12.42%)|0|2.7.1| > |B|2|In Service|5.47 TB|12.78 TB|0 B|1.46 TB|27657|2.65 TB (48.37%)|0|2.7.1| -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12806) Use yarn application -kill and kill by rm webpage the app information cannot log to userlogs directory so jobhistory cannot display it.
maobaolong created HDFS-12806: - Summary: Use yarn application -kill and kill by rm webpage the app information cannot log to userlogs directory so jobhistory cannot display it. Key: HDFS-12806 URL: https://issues.apache.org/jira/browse/HDFS-12806 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.1 Environment: Use `yarn application -kill` can successfully kill the job but the app information cannot generate into the "/userlogs/history/done_intermediate". So the jobhistory cannot display the job information. But, use `yarn application -kill` can work well. Reporter: maobaolong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12803) We should not lock FsNamesystem even we operate a sub directory, we should refinement the lock
maobaolong created HDFS-12803: - Summary: We should not lock FsNamesystem even we operate a sub directory, we should refinement the lock Key: HDFS-12803 URL: https://issues.apache.org/jira/browse/HDFS-12803 Project: Hadoop HDFS Issue Type: Bug Components: hdfs Affects Versions: 3.0.0-alpha3, 2.7.1 Reporter: maobaolong An example: If a client is doing mkdir or delete a file, other client will wait for the FSNamesystem's lock to do some operation. I think we have to refinement the lock. we can lock the parent inode only. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11752) getNonDfsUsed return 0 if reserved bigger than actualNonDfsUsed
[ https://issues.apache.org/jira/browse/HDFS-11752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong resolved HDFS-11752. --- Resolution: Not A Problem > getNonDfsUsed return 0 if reserved bigger than actualNonDfsUsed > --- > > Key: HDFS-11752 > URL: https://issues.apache.org/jira/browse/HDFS-11752 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Affects Versions: 2.7.1 >Reporter: maobaolong > Labels: datanode, hdfs > Fix For: 2.7.1 > > > {code} > public long getNonDfsUsed() throws IOException { > long actualNonDfsUsed = getActualNonDfsUsed(); > if (actualNonDfsUsed < reserved) { > return 0L; > } > return actualNonDfsUsed - reserved; > } > {code} > The code block above is the function to caculate nonDfsUsed, but in fact it > will let the result to be 0L out of expect. Such as this following situation: > du.reserved = 50G > Disk Capacity = 2048G > Disk Available = 2000G > Dfs used = 30G > usage.getUsed() = dirFile.getTotalSpace() - dirFile.getFreeSpace() > = 2048G - 2000G > = 48G > getActualNonDfsUsed = usage.getUsed() - getDfsUsed() > = 48G - 30G > = 18G > 18G < 50G, so the function `getNonDfsUsed` actualNonDfsUsed < reserved, and > the NonDfsUsed will return 0, is that logic make sense? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11752) getNonDfsUsed return 0 if reserved bigger than actualNonDfsUsed
maobaolong created HDFS-11752: - Summary: getNonDfsUsed return 0 if reserved bigger than actualNonDfsUsed Key: HDFS-11752 URL: https://issues.apache.org/jira/browse/HDFS-11752 Project: Hadoop HDFS Issue Type: Bug Components: datanode, hdfs Affects Versions: 2.7.1 Reporter: maobaolong Fix For: 2.7.1 {code} public long getNonDfsUsed() throws IOException { long actualNonDfsUsed = getActualNonDfsUsed(); if (actualNonDfsUsed < reserved) { return 0L; } return actualNonDfsUsed - reserved; } {code} The code block above is the function to caculate nonDfsUsed, but in fact it will let the result to be 0L out of expect. Such as this following situation: du.reserved = 50G Disk Capacity = 2048G Disk Available = 2000G Dfs used = 30G usage.getUsed() = dirFile.getTotalSpace() - dirFile.getFreeSpace() = 2048G - 2000G = 48G getActualNonDfsUsed = usage.getUsed() - getDfsUsed() = 48G - 30G = 18G 18G < 50G, so the function `getNonDfsUsed` actualNonDfsUsed < reserved, and the NonDfsUsed will return 0, is that logic make sense? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org