[ https://issues.apache.org/jira/browse/HDFS-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HDFS-2209: --------------------------------- Target Version/s: 0.23.0, 0.24.0 Status: Patch Available (was: In Progress) > MiniDFS cluster improvements > ---------------------------- > > Key: HDFS-2209 > URL: https://issues.apache.org/jira/browse/HDFS-2209 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test > Affects Versions: 0.20.203.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Minor > Fix For: 0.24.0 > > Attachments: HDFS-2209.patch, HDFS-2209.patch > > Original Estimate: 1h > Time Spent: 1.5h > Remaining Estimate: 2h > > I've been deploying MiniDFSCluster for some testing, and while using > it/looking through the code I made some notes of where there are issues and > improvement opportunities. This is mostly minor as its a test tool, but a > risk of synchronization problems is there and does need addressing; the rest > are all feature creep. > Field {{nameNode}} should be marked as volatile as the shutdown operation can > be in a different thread than startup. Best of all, > add synchronized methods to set and get the field, as well as shutdown. > The data dir is set from from System Properties. > {code} > base_dir = new File(System.getProperty("test.build.data", > "build/test/data"), "dfs/"); > data_dir = new File(base_dir, "data"); > {code} > This is done in {{formatDataNodeDirs()}} {{corruptBlockOnDataNode()}} and > the constructor. > Improvement: have a test property in the conf file, and only read the system > property if this is unset. This will enable > multiple MiniDFSClusters to come up in the same JVM, and handle > shutdown/startup race conditions better, and avoid the > "java.io.IOException: Cannot lock storage build/test/data/dfs/name1. The > directory is already locked." messages > Messages should log to the commons logging and not {{System.err}} and > {{System.out}}. This enables containers to catch and stream better, > and include more diagnostics such as timestamp and thread Id > Class could benefit from a method to return the FS URI, rather than just the > FS. This currently has to be worked around with some tricks involving a > cached configuration > {{waitActive()}} could get confused if "localhost" maps to an IPv6 address. > Better to ask for 127.0.0.1 as the hostname; Junit > test runs may need to be set up to force in IPv4 too. > {{injectBlocks}} has a spelling error in the IOException, > "SumulatedFSDataset" is the correct spelling -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira