Hi Alexander, Your NNBench usage looks basically correct, but NNBench is not a standard Hadoop tool. It does not implement the org.apache.hadoop.util.Tool interface, it does not execute through org.apache.hadoop.util.ToolRunner, and therefore it does not support the command line arguments that a lot of other Hadoop tools like the FsShell support. Specifically, it does not support passing -D arguments to override fs.defaultFS or any other configuration properties.
An alternative way to handle this would be to get a local copy of the configuration directory from the remote cluster that you want to test. I expect those configuration files would have fs.defaultFS set to the URL of that remote cluster in core-site.xml. Before launching NNBench, run "export HADOOP_CONF_DIR=<path to local copy of configuration files>". After exporting that environment variable, you can run "hadoop classpath" to print the classpath that will be used by all "hadoop" commands and confirm that the correct configuration directory for the target cluster is on the classpath. Then, you can run NNBench again, but drop the -D argument, since it's going to get ignored anyway. I don't see any reason why NNBench shouldn't implement the standard Tool interface and thus support the command line arguments that you were expecting. If you'd like to request that as an enhancement, please go ahead and file an HDFS jira to request it. Feel free to post a patch too if you're inclined. Otherwise, someone else in the community can pick it up. I hope this helps. --Chris Nauroth On 7/22/15, 12:41 AM, "Alexander Striffeler" <a.striffe...@students.unibe.ch> wrote: >Hi all > >I'm pretty new to the Hadoop environment and I'm about performing some >micro benchmarks. In particular, I'm struggling with executing NNBench >against an external File System: > >hadoop jar >/usr/hdp/2.2.6.0-2800/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-t >ests.jar >nnbench -Dfs.defaultFS='hfds://<external.file.system>' -operation >create_write -bytesToWrite 10 -maps 2 -reduces 1 -numberOfFiles 100 >-baseDir hdfs://dapsilon.daplab.ch/user/username/nnbench-`hostname -s` > >yields in >java.lang.IllegalArgumentException: Wrong FS: >hdfs://<external.file.system>/user/username/nnbench-hostname/data, >expected: hdfs://<native fs> > >If I neglect the ext FS prefix in the baseDir, NNBench simply ignores >the -D option and writes the files to the native DFS. Does anyone have >an idea how to solve this and nnbench an external DFS? > >Thanks a lot, any hints are very appreciated! >Regards, >Alex
