Re: is it possible to separate hdfs and mapreduce clusters

2011/4/15 changbin.wang > Hi, > I have a question that is it possible to install hdfs on physical machines > and running mapreduce on virtual machine. Possible > if possible, the performance is > acceptable or not for about 10G/hour data. It will have some lose in performance data need to be

Re: Help for the problem of running lucene on Hadoop

You'd better make the index in localfile,and copy the final index into the hdfs~~ It is not recommanded to using hdfs as the FileSystem for lucene(Though it can be used for search) 2010/12/31 Jander g > Hi, all > > I want to run lucene on Hadoop, The problem as follows: > > IndexWriter writer

Re: Namenode not starting up

did u format it? 2010/7/22 Denim Live > Hi all, > > I just restarted my cluster and now the namenode is not starting up. I get > the > following error: > > 10/07/22 09:14:40 INFO namenode.NameNode: STARTUP_MSG: > / > STARTUP_MSG: Starti

Re: how to add jar file to hadoop

try DistributedCache http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#DistributedCache 2010/7/8 Ahmad Shahzad > Hi ALL, > How can i add a jar file of my own to hadoop directory and than > call the classes that are in that jar file from hadoop classes. > > Regards, > A

Re: Does hadoop need to have ZooKeeper to work?

No, they are separate projects! they don't depend on each other~~ 2010/6/28 legolas > > Hi, > > I am wondering whether Hadoop has some dependencies on ZooKeeper or not. I mean when I download > http://apache.thelorne.com/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz > does it has ZooKeeper wit

where can we get the details of the history jobs~

We can see the details(the counters) of the resent jobs from the web but I can't found the details(the counters) of the history jobs is there any way to see them?

Re: a question about tasktracker in hadoop

Yes, u r right~~ 2010/3/22 毛宏 > I read from 《Towards Optimizing Hadoop Provisioning in the Cloud 》 > saying that "mapred.tasktracker.map.tasks.maximum and > mapred.tasktracker.reduce.tasks.maximum respectively set the maximum > number of parallel mappers and reducers that can run on a Hadoop > s

Re: Namenode problem

) > at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446) > > 2010-03-09 22:45:55,334 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: > / > SHUTDOWN_MSG: Shutting down DataNode at weliam-d

Re: Namenode problem

It's usually in $HADOOP_HOME/logs 2010/3/9 William Kang > Hi, > If the namenode is not up, how can I get the logdir? > > > William > > On Mon, Mar 8, 2010 at 10:39 PM, Eason.Lee wrote: > > > 2010/3/9 William Kang > > > > > Hi Eason, > &

Re: Namenode problem

gt; format will delete the old namespace, so everything will lost~~ > > > William > > On Mon, Mar 8, 2010 at 1:02 AM, Eason.Lee wrote: > > > 2010/3/8 William Kang > > > > > Hi guys, > > > Thanks for your replies. I did not put anything in /tmp. It's ju

Re: Namenode problem

really > see > anything in log. Not sure what caused it. > > > William > > > On Mon, Mar 8, 2010 at 12:39 AM, Bradford Stephens < > bradfordsteph...@gmail.com> wrote: > > > Yeah. Don't put things in /tmp. That's unpleasant in the long run. > &

Re: Namenode problem

Your /tmp directory is cleaned automaticly? Try to set dfs.name.dir/dfs.data.dir to a safe dir~~ 2010/3/8 William Kang > Hi all, > I am running HDFS in Pseudo-distributed mode. Every time after I restarted > the machine, I have to format the namenode otherwise the localhost:50070 > wont show up

Re: do all mappers finish before reducer starts

No,Reduce will start as soon as Map starts So reduce can start transfer map outputs to local when some of they are finished 2010/1/27 adeelmahmood > > I just have a conceptual question. My understanding is that all the mappers > have to complete their job for the reducers to start working becaus

Re: Problem with fuse-dfs~

Yes~ It works! Thanks very much~~~ 2010/1/22 Michael Thomas > On 01/21/2010 09:37 PM, Eason.Lee wrote: > >> I'am trying to mount hdfs to my system using fuse-dfs >> hadoop:hadoop-0.20.1+152 >> fuse:fuse-2.8.1 >> os:SUSE Linux Enterprise Server 10 SP2 (x86_64)

Problem with fuse-dfs~

I'am trying to mount hdfs to my system using fuse-dfs hadoop:hadoop-0.20.1+152 fuse:fuse-2.8.1 os:SUSE Linux Enterprise Server 10 SP2 (x86_64) I compile fuse-dfs like this: ant compile -Dcompile.c++=true -Dlibhdfs=true ln -s $HADOOP_HOME/build/c++/Linux-amd64-64/lib $HADOOP_HOME/build/libhdfs exp

Error while building project(common) with ant

The error shows that something wrong with package-info.java compile-core-classes: [javac] Compiling 346 source files to E:\projects\HadoopCommon\ build\classes [javac] E:\projects\HadoopCommon\build\src\org\apache\hadoop\package-info.java:5: 未结束的字符串字面值 [javac] u

Error while building project(common) with ant

The error shows that something wrong with package-info.java compile-core-classes: [javac] Compiling 346 source files to E:\projects\HadoopCommon\build\classes [javac] E:\projects\HadoopCommon\build\src\org\apache\hadoop\package-info.java:5: 未结束的字符串字面值 [javac] us

Re: Lucene + Hadoop

I think you'd better using map to group all the file belong to the same author together and using reduce to index the files? 2009/11/11 Otis Gospodnetic > I think that sounds right. > I believe that's what I did when I implemented this type of functionality > for http://simpy.com/ > > I'm not su

Re: Error register getProtocolVersion

2009-10-14 11:09:54,232 INFO org.apache.hadoop.ipc.Server: Error register getProtocolVersion java.lang.IllegalArgumentException: Duplicate metricsName:getProtocolVersion at org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:53) at org.apache.hadoop.metrics.util.Me

Re: help with Hadoop custom Writable types implementation

I think SubStorage should implements Writable as well~~ Or You can try ObjectWritable 2009/10/15 z3r0c001 > I'm trying to implement Writable interface. but not sure how to > serialize/write/read data from nested objects in > > public class StorageClass implements Writable{ > > public String xS

Re: fuse:The ratio of reported blocks 0.0000 has not reached the threshold 0.9990

seams no datanode is coming~~no block is reported, so hdfs is in safemode check if datanode is up, and network between namenode and datanode is fine~~ 2009/10/13 yibo820217 > > hi,recently,i got some problem. > at first,I start the hadoop > #bin/start-all.sh > then,I mount hdfs to local > #fuse_

Re: How to deal with safemode?

now which block is lost... I was doing nothing when the network went wrong~ I have install hbase on the hdfs Does that means I have lost the data in the hbase? > > On 10/12/09, Eason.Lee wrote: > > thx for reply~~ > > more info show by fsck > > > > Status: HEALTHY >

Re: How to deal with safemode?

i restart the cluster, it still can't leave the safemode 2. Check if all your datanodes are coming up. Could be that there's > some DN that isn't coming up - causing the under reporting of blocks. > > all the datanode is coming up I have only 4 datanode > On 10/1

How to deal with safemode?

There is something wrong with network, so i killed all the hadoop thread buy "kill -9 pid" when i try to start hadoop today, it can't leave safemode automatically! the web ui shows: *Safe mode is ON. The ratio of reported blocks 0.9951 has not reached the threshold 0.9990. Safe mode will be turned

Re: How to start a datanode?

e working cluster > just > like any other node. > > > Tamir > > > On Mon, Oct 12, 2009 at 11:44 AM, Eason.Lee wrote: > > > One of my datanode is stop > > Can I start the datanode and add it to the cluster without restarting the > > hole cluster? > > >

Re: wiki home page

OH MY GOD! 2009/9/22 Y G > hi all: > Someone modified wiki home page and left some “strange” remarks.( > http://wiki.apache.org/hadoop) > > here is the remark: > > > Big bug in Hadoop MapReduce !!! > When > > I use too Many Counters in a big job(processin

Re: Job Failue, Run it again? Or anything else to do?

[?]，but thanks all the same! 2009/9/18 Harish Mallipeddi > On Fri, Sep 18, 2009 at 6:19 AM, Eason.Lee wrote: > > > I have run my job over night. > > But i found that 4 of the maps failed for "java.lang.OutOfMemoryError: > Java > > heap space" exception. &

Job Failue, Run it again? Or anything else to do?

I have run my job over night. But i found that 4 of the maps failed for "java.lang.OutOfMemoryError: Java heap space" exception. Do I have to run all the job again? Or anything else I can do with it?

Re: How to report the status

sure if what you are experiencing is similar. > Context.progress() "should" work. One ugly hack would be to set the timeout > value to high number. But I would wait for a better answer before doing > that. > > Thanks, > Amogh > > -Original Message- > From: Eason.

How to report the status

I am using hadoop Map job to import data into hbase from DB For there is more than 10M records.It takes more than 10 minutes to finish one map job. So I got "failed to report status for 604 seconds" Error! I have set in my Map as followed: cnkiTable.put(put); if ((++count % 1000)==0) { contex

Re: Can Namenode and JobTracker run in different server?

r2: > > bin/start-mapred.sh > > Hope this helps. > > Tom > > On Tue, Jun 30, 2009 at 7:53 AM, Eason.Lee wrote: > > Just want to run NameNode and JobTracker in separate server,for example > > run NameNode in server1 and JobTracker in server2 > > > &g

Can Namenode and JobTracker run in different server?

Just want to run NameNode and JobTracker in separate server,for example run NameNode in server1 and JobTracker in server2 my config is just like this, but It seems jobtracker not start correctly! fs.default.name hdfs://server1:9000/ mapred.job.tracker server2:9