Re: local node Quotas (for an R&D cluster)

Allen Wittenauer wrote: On 9/22/09 5:47 PM, "Ravi Phulari" wrote: Hello Paul here is quick answer to your question - You can use dfs.datanode.du.pct and dfs.datanode.du.reserved property in hdfs-site.xml config file to configure maximum local disk space used by hdfs and mapreduce.

Re: local node Quotas (for an R&D cluster)

On Tue, Sep 22, 2009 at 9:40 PM, Allen Wittenauer wrote: > > > > On 9/22/09 5:47 PM, "Ravi Phulari" wrote: > >> Hello Paul here is quick answer to your question - >> You can use dfs.datanode.du.pct  and dfs.datanode.du.reserved  property in >> hdfs-site.xml config file to  configure >> maximum  l

Re: local node Quotas (for an R&D cluster)

On 9/22/09 5:47 PM, "Ravi Phulari" wrote: > Hello Paul here is quick answer to your question - > You can use dfs.datanode.du.pct and dfs.datanode.du.reserved property in > hdfs-site.xml config file to configure > maximum local disk space used by hdfs and mapreduce. No, that's incorrect.

Re: local node Quotas (for an R&D cluster)

Paul , Previous versions of Hadoop also supported these configuration options. Yes , your understanding of dfs.datanode.du.pct and dfs.datanode.du.reserved is correct . Link mentioned in previous email talks about configuration files in general. For detailed information about each configurations

Re: local node Quotas (for an R&D cluster)

ok, that's great, thanks so much there, there are so many options in hadoop it's a bit hard to see the forest for the trees! (actually, honestly if those default files were sorted by keys I think it would make it easier for people to find) However when I go to the link provided it's not su

Re: local node Quotas (for an R&D cluster)

Hello Paul here is quick answer to your question - You can use dfs.datanode.du.pct and dfs.datanode.du.reserved property in hdfs-site.xml config file to configure maximum local disk space used by hdfs and mapreduce. dfs.datanode.du.pct 0.85f When calculating remaining space, onl

local node Quotas (for an R&D cluster)

Hi, I recognize that Hadoop has built in quotas for directories inside HDFS, and that one can configure the 'dfs.data.dir' property to specify the paths to use on a local node for DFS blocks, but I have a couple of questions regarding setting up a trial Hadoop cluster for R&D purposes that

Re: Prepare input data for Hadoop

Use an external database (e.g., mysql) or some other transactional bookkeeping system to record the state of all your datasets (STAGING, UPLOADED, PROCESSED) - Aaron On Thu, Sep 17, 2009 at 7:17 PM, Huy Phan wrote: > Hi all, > > I have a question about strategy to prepare data for Hadoop to ru

Re: SequenceFileAsBinaryOutputFormat for M/R

In the 0.20 branch, the common best-practice is to use the old API and ignore deprecation warnings. When you get to 0.22, you'll need to convert all your code to use the new API. There may be a new-API equivalent in org.apache.hadoop.mapreduce.lib.output that you could use, if you convert your Map

Re: RandomAccessFile with HDFS

Or maybe more pessimistically, the second "stable" append implementation. It's not like HADOOP-1700 wasn't intended to work. It was just found not to after the fact. Hopefully this reimplementation will succeed. If you're running a cluster that contains mission-critical data that cannot tolerate c

Re: wiki home page

Now it is back to normal － 天天开心 身体健康 2009/9/22 Eason.Lee > OH MY GOD! > > 2009/9/22 Y G > > > hi all: > > Someone modified wiki home page and left some “strange” remarks.( > > http://wiki.apache.org/hadoop) > > > > here is the remark: > > > > > Big bug in Hadoop MapReduce

Apache Hadoop Get Together: Next week Tuesday, newthinking store Berlin Germany

This is a friendly reminder that the next Apache Hadoop Get Together takes place next week on Tuesday, 29th of September* at newthinking store (Tucholskystr. 48, Berlin): http://upcoming.yahoo.com/event/4314020/ * Thorsten Schuett, Solving Puzzles with MapReduce. * Thilo Götz, Text analyti

Re: wiki home page

OH MY GOD! 2009/9/22 Y G > hi all: > Someone modified wiki home page and left some “strange” remarks.( > http://wiki.apache.org/hadoop) > > here is the remark: > > > Big bug in Hadoop MapReduce !!! > When > > I use too Many Counters in a big job(processin

wiki home page

hi all: Someone modified wiki home page and left some “strange” remarks.( http://wiki.apache.org/hadoop) here is the remark: > Big bug in Hadoop MapReduce !!! When > I use too Many Counters in a big job(processing about 4T data, 1 billion > record), I ofte