ashishkumar50 commented on code in PR #8365: URL: https://github.com/apache/ozone/pull/8365#discussion_r2151321917
########## hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/DUOptimized.java: ########## @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.hdds.fs; + +import java.io.File; +import java.util.function.Supplier; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Make use of DU class that uses the unix 'du' program to calculate space usage of metadata excluding container data. + * Container data usages is identified from container set. + * + * @see SpaceUsageSource + */ +public class DUOptimized implements SpaceUsageSource { + private static final Logger LOG = LoggerFactory.getLogger(DUOptimized.class); + + private final DU metaPathDU; + private Supplier<Supplier<Long>> containerUsedSpaceProvider; + + public DUOptimized(File path, Supplier<File> exclusionProvider) { + metaPathDU = new DU(exclusionProvider, path); + } + + @Override + public long getUsedSpace() { + long metaPathSize = metaPathDU.getUsedSpace(); + if (null == containerUsedSpaceProvider) { + return metaPathSize; + } + Supplier<Long> gatherContainerUsages = containerUsedSpaceProvider.get(); + long containerUsedSpace = gatherContainerUsages.get(); + LOG.info("Disk du usages {}, container data usages {}", metaPathSize, containerUsedSpace); Review Comment: ```suggestion LOG.info("Disk metaPath du usages {}, container data usages {}", metaPathSize, containerUsedSpace); ``` ########## hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/HddsVolume.java: ########## @@ -405,6 +408,38 @@ public long getFreeSpaceToSpare(long volumeCapacity) { return getDatanodeConfig().getMinFreeSpace(volumeCapacity); } + @Override + public void setGatherContainerUsages(Function<HddsVolume, Long> gatherContainerUsages) { + this.gatherContainerUsages = gatherContainerUsages; + } + + @Override + protected long containerUsedSpace() { + return gatherContainerUsages.apply(this); + } + + @Override + public File getContainerDirsPath() { + if (getStorageState() != VolumeState.NORMAL) { + return null; + } + File hddsVolumeRootDir = getHddsRootDir(); + //filtering storage directory + File[] storageDirs = hddsVolumeRootDir.listFiles(File::isDirectory); + if (storageDirs == null) { + LOG.error("IO error for the volume {}, directory not found", hddsVolumeRootDir); + return null; + } + File clusterIDDir = new File(hddsVolumeRootDir, getClusterID()); + if (storageDirs.length == 1 && !clusterIDDir.exists()) { + // If this volume was formatted pre SCM HA, this will be the SCM ID. + // A cluster ID symlink will exist in this case only if this cluster is finalized for SCM HA. + // If the volume was formatted post SCM HA, this will be the cluster ID. + clusterIDDir = storageDirs[0]; + } + return new File(clusterIDDir, Storage.STORAGE_DIR_CURRENT); Review Comment: What happens if `clusterIDDir` doesn't exist and `storageDirs.length != 1`? ########## hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/DU.java: ########## @@ -96,11 +106,21 @@ public long getUsed() { @Override public String toString() { + if (exclusionProvider != null) { + return String.join(" ", getExecString()) + "\n" + value.get() + "\t" + getPath(); + } Review Comment: No need to have condition here as `getExecString` already handles with and without `exclusionProvider.` ########## hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/SpaceUsageCheckParams.java: ########## @@ -19,6 +19,7 @@ import static com.google.common.base.Preconditions.checkArgument; +import com.google.common.base.Supplier; Review Comment: ```suggestion import java.util.function.Supplier; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
