Re: Any samples of how to write a custom FileSystem

Steven, You could also look at HttpFSFilesystem in the hadoop-httpfs module, it is quite simple and selfcontained. Cheers. Alejandro On Tue, Jan 31, 2012 at 8:37 PM, Harsh J wrote: > To write a custom filesystem, extend on the FileSystem class. > > Depending on the scheme it is supposed to se

Re: Any samples of how to write a custom FileSystem

To write a custom filesystem, extend on the FileSystem class. Depending on the scheme it is supposed to serve, creating an entry "fs..impl" in core-site.xml, and then loading it via the FileSystem.get(URI, conf) API will auto load it for you, provided the URI you pass has the right scheme. So sup

Re: plugging in to hadoop

I am actually looking at implementing my own UI. I am looking to develop monitoring and control framework for hadoop I am mainly looking at hadoo 0.23 On 2/1/2012 8:36 AM, real great.. wrote: I think there are a couple of projects already looking into UI. did you check them? On Wed, Feb 1, 201

Re: plugging in to hadoop

I think there are a couple of projects already looking into UI. did you check them? On Wed, Feb 1, 2012 at 7:57 AM, tejas wrote: > Hi > > I am trying to develop a UI and display metrics of mapreduce and HDFS and > also provide some kind of monitoring. > > 1. With version 0.20, I was able to exten

Re: Does Hadoop 0.20.205 and Ganglia 3.1.7 compatible with each other ?

I would be glad to hear that too.. I've setup the following: Hadoop 0.20.205 Ganglia Front 3.1.7 Ganglia Back *(gmetad)* 3.1.7 RRDTool 1.4.5. -> i had some troubles installing 1.4.4 Ganglia works just in case hadoop is not running, so metrics are not publshed to gmetad

metrics2 not working

Hi I set up hadoop 0.23. I configured hadoop-metrics.properties as well as hadoop-metrics2.properties to dump metric files. However i see that hadoop still does not pick the metrics2 property file and still logs as per the configurations in hadoop-metrics-properties Is there some specific con

plugging in to hadoop

Hi I am trying to develop a UI and display metrics of mapreduce and HDFS and also provide some kind of monitoring. 1. With version 0.20, I was able to extend the org.apache.JobTrackerPlugin class by configuring it in the mapred-site.properties. Is it possible to make a similar configuration

Re: Adding mahout math jar to hadoop mapreduce execution

You also need to add the jar to the classpath so it's available in your main. You can do soemthing like this: HADOOP_CLASSPATH=/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar hadoop jar ... -Joey On Tue, Jan 31, 2012 at 1:38 PM, Daniel Quach wrote: > For Hadoop 0.20.203 (the latest s

Re: Namenode service not running on the Configured IP address

Hi Harsh/Praveenesh, Thanks for the reply guys. I forgot to mention that /etc/hosts was having the ip to dns mapping and Yes, i deliberately used IP address in the configuration because i thought that if i use host-name then how would hadoop know about the network interface on which it is suppose

Re: Best practices for hadoop shuffling/tunning ?

Moving to mapreduce-user@, bcc common-user@. Please use project specific lists. Your io.sort.mb is too high. You only have 1G of heap for the map. Reduce parallel copies is too high too. On Jan 30, 2012, at 4:50 AM, praveenesh kumar wrote: > Hey guys, > > Just wanted to ask, are there any sort

S3 block file system

Hello, Is S3 block file system still supported ? I read in aws docs that s3 block file system is deprecated; Not sure if the doc is in error (or old) Thanks, Madhu

Re: Adding mahout math jar to hadoop mapreduce execution

For Hadoop 0.20.203 (the latest stable), is it sufficient to do this to parse the lib jars from the command line? public static main (String args[]) { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); Job job = new Job(conf, "

Re: Hybrid Hadoop with fork/join ?

Rob, Hadoop has as a way to run Map tasks in multithreading mode, look for the MultithreadedMapRunner & MultithreadedMapper. Thanks. Alejandro. On Tue, Jan 31, 2012 at 7:51 AM, Rob Stewart wrote: > Hi, > > I'm investigating the feasibility of a hybrid approach to parallel > programming, by fu

Re: Hadoop Datacenter Setup

That was another one of our options (PXE/NFS netboot). I'm just afraid of NFS locking on me at random times. If you've had any success with running a live read-only root filesystem off of NFS, I'd be more compelled in using it. If our image is small enough (~500 MB), it may make sense to per

Re: Hadoop Datacenter Setup

Hey Patrick, thanks for your reply. I'm still skeptical when it comes to flash drives, especially as pertains to Hadoop. The write cycle limit is impractical to make them usable for dfs.data.dir and mapred.local.dir, and as you pointed out, you can't use them for logs either. I was going to use t

Hybrid Hadoop with fork/join ?

Hi, I'm investigating the feasibility of a hybrid approach to parallel programming, by fusing together the concurrent Java fork/join libraries with Hadoop... MapReduce, a paradigm suited for scalable execution over distributed memory + fork/join, a paradigm for optimal multi-threaded shared memory

Re: refresh namenode topology cache

We are using script based mapping and got the order wrong for a couple of nodes, will look into custom mapping class moving fwd. Thanks On Tue, Jan 31, 2012 at 12:42 AM, Harsh J wrote: > Sateesh, > > On Tue, Jan 31, 2012 at 12:24 AM, Sateesh Lakkarsu > wrote: > > A couple of nodes got registere

Re: Best practices for hadoop shuffling/tunning ?

Can anyone please eyeball the config parameters as defined below and share their thoughts on this ? Thanks, Praveenesh On Mon, Jan 30, 2012 at 6:20 PM, praveenesh kumar wrote: > Hey guys, > > Just wanted to ask, are there any sort of best practices to be followed > for hadoop shuffling improveme