Setting HDFS directory time programmatically

Hi - Is it possible to set the access time of a HDFS directory programmatically? I’m using 0.20.204.0. I need to do that in unit tests, where my clean up program is going to remove files/dirs whose access time is too far in the past. I can setTimes on the test files without any problem, but no

Re: Question about accessing another HDFS

ith B.company in it) using Configuration, > grab a FileSystem obj with that and then go forward. Seems like some > unnecessary overhead though. > > Thanks, > > Tom > > On Thu, Dec 8, 2011 at 2:42 PM, Frank Astier > wrote: > > Hi - > > > > W

Question about accessing another HDFS

Hi - We have two namenodes set up at our company, say: hdfs://A.mycompany.com hdfs://B.mycompany.com >From the command line, I can do: Hadoop fs –ls hdfs://A.mycompany.com//some-dir And Hadoop fs –ls hdfs://B.mycompany.com//some-other-dir I’m now trying to do the same from a Java program tha

Hadoop start up error

Hi - I’m seeing the following exception while trying to start MiniDFSCluster in some environment. The stack trace is: Starting DataNode 0 with dfs.data.dir: build/test/data/dfs/data/data1,build/test/data/dfs/data/data2 java.lang.NullPointerException at org.apache.hadoop.hdfs.MiniDFSC

Question about superuser and permissions

Hi - I’m writing unit tests that programatically start a name node and populate HDFS directories, but I want to simulate the situation where I don’t have read access to some HDFS directory (which happens on the real grid I eventually deploy to). I’ve tried to chown and chmod, but it seems to h

Permissions when creating a file

Hi - I’m stumped on the following problem: I’m creating a FSDataOutputStream with some permissions, write to it, close it, and low and behold the permissions don’t seem to be what I set!! Here is the whole program and its output: Configuration xconf = new Configuration();

Debugging mapper

Hi - I’m using IntelliJ and the WordCount example in Hadoop (which uses MiniMRCluster). Is it possible to set an IntelliJ debugger breakpoint straight into the map function of the mapper? - I’ve tried, but so far, the debugger does not stop at the breakpoint. Thanks! Frank

Turn off all Hadoop logs?

Is it possible to turn off all the Hadoop logs simultaneously? In my unit tests, I don’t want to see the myriad “INFO” logs spewed out by various Hadoop components. I’m using: ((Log4JLogger) DataNode.LOG).getLogger().setLevel(Level.OFF); ((Log4JLogger) LeaseManager.LOG).getLogger().setLeve

Error with logging in (my) unit tests

Hi - I’m working with Maven inside IntelliJ, using Hadoop 0.20.203.0, and I get the following error message when trying to run my own unit tests, that use MiniDFSCluster. I copied how to use MiniDFSCluster from the Hadoop unit tests. The message: log4j:ERROR Could not instantiate class [org.a

Hadoop in process?

Hi - Is there a way I can start HDFS (the namenode) from a Java main and run unit tests against that? I need to integrate my Java/HDFS program into unit tests, and the unit test machine might not have Hadoop installed. I’m currently running the unit tests by hand with hadoop jar ... My unit tes