Mingliang Liu created HDFS-9027: ----------------------------------- Summary: Refactor o.a.h.hdfs.DataStreamer$isLazyPersist() method Key: HDFS-9027 URL: https://issues.apache.org/jira/browse/HDFS-9027 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Mingliang Liu Assignee: Mingliang Liu
In method {{isLazyPersist()}}, the {{org.apache.hadoop.hdfs.DataStreamer}} class checks whether the HDFS file is lazy persist. It does two things: 1. Create a class-wide _static_ {{BlockStoragePolicySuite}} object, which builds an array of {{BlockStoragePolicy}} internally 2. Get a block storage policy object from the {{blockStoragePolicySuite}} by policy name {{HdfsConstants.MEMORY_STORAGE_POLICY_NAME}} Code samples as following: {code} private static final BlockStoragePolicySuite blockStoragePolicySuite = BlockStoragePolicySuite.createDefaultSuite(); static boolean isLazyPersist(HdfsFileStatus stat) { final BlockStoragePolicy p = blockStoragePolicySuite.getPolicy( HdfsConstants.MEMORY_STORAGE_POLICY_NAME); return p != null && stat.getStoragePolicy() == p.getId(); } {code} This has two side effects: 1. Takes time to iterate the pre-built block storage policy array in order to find the _same_ policy every time whose id matters only (as we need to compare the file status policy id with lazy persist policy id) 2. {{DataStreamer}} class imports {{BlockStoragePolicySuite}}. The former should be moved to {{hadoop-hdfs-client}} module, while the latter can stay in {{hadoop-hdfs}} module. Actually, we have the block storage policy IDs, which can be used to compare with HDFS file status' policy id, as following: {code} static boolean isLazyPersist(HdfsFileStatus stat) { return stat.getStoragePolicy() == HdfsConstants.MEMORY_STORAGE_POLICY_ID; } {code} This way, we only need to move the block storage policies' IDs from {{HdfsServerConstant}} ({{hadoop-hdfs}} module) to {{HdfsConstants}} ({{hadoop-hdfs-client}} module). Another reason we should move those block storage policy IDs is that the block storage policy names were moved to {{HdfsConstants}} already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)