Re: Hadoop - Eclipse Plugin Error
Srinivas, Hadoop is running in localhost :(. On Sat, Jan 22, 2011 at 11:23 AM, Raghu R wrote: > Mike, > > Yes I am running a pseudo cluster in localhost. Hadoop is running from > 'hadoop' user group, however eclipse is running from 'raghu' user group. In > eclipse plugin configuration I have mentioned the Username as 'hadoop'. > Please tell me why HMaster works with port 9000, though the default > port is 54310 per configuration. > > > On Sat, Jan 22, 2011 at 10:31 AM, Srinivas Pinisetti < > srinivas.pinise...@gmail.com> wrote: > >> Hi Raghu, >> >> You may want to make sure you are having the same JAR versions >> (hadoop,hbase,zookeeper) on both Client & Server. >> >> Thanks, >> Srinivas Pinisetti >> >> On Wed, Jan 19, 2011 at 1:04 AM, Segel, Mike wrote: >> >>> Raghu, >>> >>> Are you running your cluster on localhost? >>> (Meaning are you running a pseudo cluster on the same machine as your >>> eclipse session? ) >>> >>> HTH >>> -Mike >>> >>> >>> >>> >>> -Original Message- >>> From: Raghu R [mailto:raghu@gmail.com] >>> Sent: Tuesday, January 18, 2011 11:56 AM >>> To: common-user@hadoop.apache.org; common-...@hadoop.apache.org >>> Subject: Hadoop - Eclipse Plugin Error >>> >>> Hi Hadoopers, >>> >>> I am trying to connect hadoop from eclipse to run MapReduce jobs. >>> I am getting the following error while connecting to the DFS Locations in >>> eclipse, >>> >>> "Error: Call to localhost/127.0.0.1:54310 failed on local exception: >>> java.io.EOFException" >>> >>> I get the "Can not start master" error while trying to connect HBase and >>> Hadoop because of the same exception. >>> >>> >>> Please let me know what could be wrong with the configuration, >>> >>> The current config for your reference, >>> >>> core-site.xml [hadoop conf] >>> - >>> >>> fs.default.name >>> hdfs://localhost:54310 >>> . >>> >>> >>> hbase-site.xml [HBase conf] >>> >>> >>> hbase.rootdir >>> hdfs://localhost:54310/hbase >>> >>> >>>hbase.cluster.distributed >>>true >>> >>> >>> Eclispe Configuration: >>> -- >>> DFS Master >>> Host: localhost >>> Port: 54310 >>> >>> >>> Hadoop Version: 0.21.0 >>> HBase Version: 0.20.6 >>> Eclipse Plugin: hadoop-eclipse-plugin-0.20.3-SNAPSHOT.jar >>> >>> >>> >>> Regards, >>> Raghu :) >>> >>> >>> The information contained in this communication may be CONFIDENTIAL and >>> is intended only for the use of the recipient(s) named above. If you are >>> not the intended recipient, you are hereby notified that any dissemination, >>> distribution, or copying of this communication, or any of its contents, is >>> strictly prohibited. If you have received this communication in error, >>> please notify the sender and delete/destroy the original message and any >>> copy of it from your computer or paper files. >>> >> >> >> >> -- >> Thanks, >> Srinivas >> > >
Re: When applying a patch, which attachment should I use?
Dear Sharma, I am not a Zookeeper professional since this is the first time I have installed zookeeper myself. But looking at your log, I think the problem is either with your firewall setting, or server connection setting. About the patch installation, download the patch you'd like to apply and place them in $(HADOOP_HOME) then apply the patch by typing: patch -p0 < "patch_name" Good luck Regards, Ed 2011/1/21 Adarsh Sharma > Extremely Sorry, Forgot to attach logs : > Here they are : > > > Adarsh Sharma wrote: > >> Thanx Edward, Today I look upon your considerations and start working : >> >> edward choi wrote: >> >>> Dear Adarsh, >>> >>> I have a single machine running Namenode/JobTracker/Hbase Master. >>> There are 17 machines running Datanode/TaskTracker >>> Among those 17 machines, 14 are running Hbase Regionservers. >>> The other 3 machines are running Zookeeper. >>> >>> >> >> I have 10 servers and a single machine running Namenode/JobTracker/Hbase >> Master. >> There are 9 machines running Datanode/TaskTracker >> Among those 9 machines, 6 are running Hbase Regionservers. >> The other 3 machines are running Zookeeper. >> I'm using hadoop-0.20.2, hbase-0.20.3 >> >>> And about the Zookeeper, >>> Hbase comes with its own Zookeeper so you don't need to install a new >>> Zookeeper. (except for the special occasion, which I'll explain later) >>> I assigned 14 machines as regionservers using >>> "$HBASE_HOME/conf/regionservers". >>> I assigned 3 machines as Zookeeperss using "hbase.zookeeper.quorum" >>> property >>> in "$HBASE_HOME/conf/hbase-site.xml". >>> Don't forget to set "export HBASE_MANAGES_ZK=true" >>> >>> >> >> I think bydefault it takes true anyways I set "export >> HBASE_MANAGES_ZK=true" in hbase-env.sh >> >> in "$HBASE_HOME/conf/hbase-env.sh". (This is where you announce that you >>> will be using Zookeeper that comes with HBase) >>> This way, when you execute "$HBASE_HOME/bin/start-hbase.sh", HBase will >>> automatically start Zookeeper first, then start HBase daemons. >>> >>> >> But perhaps I found my Hbase Master is running through Web UI. But there >> are exceptions in my Zookeeper Logs. I am also able to create table in hbase >> and view it. >> >> The onle thing I don't do is apply the *hdfs-630-0.20-append.patch* to >> each hadoop package in each node. As I don't know how to apply it. >> >> If this is the problem Please guide me the steps to apply it. >> >> I also attached my Zookeeper Logs of my Zookeeper Servers. >> Please find the attachment. >> >> Also, you can install your own Zookeeper and tell HBase to use it instead >>> of >>> its own. >>> I read it on the internet that Zookeeper that comes with HBase does not >>> work >>> properly on Windows 7 64bit. ( >>> http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/) >>> So in that case you need to install your own Zookeeper, set it up >>> properly, >>> and tell HBase to use it instead of its own. >>> All you need to do is configure zoo.cfg and add it to the HBase >>> CLASSPATH. >>> And don't forget to set "export HBASE_MANAGES_ZK=false" >>> in "$HBASE_HOME/conf/hbase-env.sh". >>> This way, HBase will not start Zookeeper automatically. >>> >>> About the separation of Zookeepers from regionservers, >>> Yes, it is recommended to separate Zookeepers from regionservers. >>> But that won't be necessary unless your clusters are very heavily loaded. >>> They also suggest that you give Zookeeper its own hard disk. But I >>> haven't >>> done that myself yet. (Hard disks cost money you know) >>> So I'd say your cluster seems fine. >>> But when you want to expand your cluster, you'd need some changes. I >>> suggest >>> you take a look at "Hadoop: The Definitive Guide". >>> >>> >>> >> >> Thanks & Best Regards >> >> Adarsh Sharma >> >> Regards, >>> Edward >>> >>> >>> >> >> >> 2011/1/13 Adarsh Sharma >>> >>> >>> Thanks Edward, Can you describe me the architecture used in your configuration. Fore.g I have a cluster of 10 servers and 1 node act as ( Namenode, Jobtracker, Hmaster ). Remainning 9 nodes act as ( Slaves, datanodes, Tasktracker, Hregionservers ). Among these 9 nodes I also set 3 nodes in zookeeper.quorum.property. I want to know that is it necessary to configure zookeeper separately with the zookeeper-3.2.2 package or just have some IP's listed in zookeeper.quorum.property and Hbase take care of it. Can we specify IP's of Hregionservers used before as zookeeper servers ( HQuorumPeer ) or we must need separate servers for it. My problem arises in running zookeeper. My Hbase is up and running in fully distributed mode too. With Best Regards Adarsh Sharma edward choi wrote: > Dear Adarsh, > > My situation is somewhat different from yours as I am only running > Hadoop > and Hbase (as opposed to Had
Re: Hadoop - Eclipse Plugin Error
Mike, Yes I am running a pseudo cluster in localhost. Hadoop is running from 'hadoop' user group, however eclipse is running from 'raghu' user group. In eclipse plugin configuration I have mentioned the Username as 'hadoop'. Please tell me why HMaster works with port 9000, though the default port is 54310 per configuration. On Sat, Jan 22, 2011 at 10:31 AM, Srinivas Pinisetti < srinivas.pinise...@gmail.com> wrote: > Hi Raghu, > > You may want to make sure you are having the same JAR versions > (hadoop,hbase,zookeeper) on both Client & Server. > > Thanks, > Srinivas Pinisetti > > On Wed, Jan 19, 2011 at 1:04 AM, Segel, Mike wrote: > >> Raghu, >> >> Are you running your cluster on localhost? >> (Meaning are you running a pseudo cluster on the same machine as your >> eclipse session? ) >> >> HTH >> -Mike >> >> >> >> >> -Original Message- >> From: Raghu R [mailto:raghu@gmail.com] >> Sent: Tuesday, January 18, 2011 11:56 AM >> To: common-user@hadoop.apache.org; common-...@hadoop.apache.org >> Subject: Hadoop - Eclipse Plugin Error >> >> Hi Hadoopers, >> >> I am trying to connect hadoop from eclipse to run MapReduce jobs. >> I am getting the following error while connecting to the DFS Locations in >> eclipse, >> >> "Error: Call to localhost/127.0.0.1:54310 failed on local exception: >> java.io.EOFException" >> >> I get the "Can not start master" error while trying to connect HBase and >> Hadoop because of the same exception. >> >> >> Please let me know what could be wrong with the configuration, >> >> The current config for your reference, >> >> core-site.xml [hadoop conf] >> - >> >> fs.default.name >> hdfs://localhost:54310 >> . >> >> >> hbase-site.xml [HBase conf] >> >> >> hbase.rootdir >> hdfs://localhost:54310/hbase >> >> >>hbase.cluster.distributed >>true >> >> >> Eclispe Configuration: >> -- >> DFS Master >> Host: localhost >> Port: 54310 >> >> >> Hadoop Version: 0.21.0 >> HBase Version: 0.20.6 >> Eclipse Plugin: hadoop-eclipse-plugin-0.20.3-SNAPSHOT.jar >> >> >> >> Regards, >> Raghu :) >> >> >> The information contained in this communication may be CONFIDENTIAL and is >> intended only for the use of the recipient(s) named above. If you are not >> the intended recipient, you are hereby notified that any dissemination, >> distribution, or copying of this communication, or any of its contents, is >> strictly prohibited. If you have received this communication in error, >> please notify the sender and delete/destroy the original message and any >> copy of it from your computer or paper files. >> > > > > -- > Thanks, > Srinivas >
Re: Losing Records with Block Compressed Sequence File
That is what I was thinking. I am using Flume for log collection/aggregation. I'll have a look at the code to see what is going on, thanks. On Fri, Jan 21, 2011 at 6:43 PM, Alan Malloy wrote: > Make sure to close the output writer? I had similar problems in a different > scenario and it turned out I was neglecting to close/flush my output. > > > On 01/21/2011 01:04 PM, David Sinclair wrote: > >> Hi, I am seeing an odd problem when writing block compressed sequence >> files. >> If I write 400,000 records into a sequence file w/o compression, all 400K >> end up in the file. If I write with block, regardless if it is bz2 or >> deflate, I start losing records. Not a ton, but a couple hundred. >> >> Here are the exact numbers >> >> bz2 399,734 >> deflate 399,770 >> none 400,000 >> >> Conf settings >> io.file.buffer.size - 4K, io.seqfile.compress.blocksize - 1MB >> >> anyone ever see this behavior? >> >> thanks >> >> dave >> >>
Re: error compiling hadoop-mapreduce
Bcc'ing common-user, adding mapreduce-user@ list instead. You have a
better chance to get your question answered if you send it to the
correct list.
For the answer see https://issues.apache.org/jira/browse/MAPREDUCE-2282
--
Take care,
Konstantin (Cos) Boudnik
On Fri, Jan 21, 2011 at 09:08, Edson Ramiro wrote:
> Hi all,
>
> I'm compiling hadoop from git using these instructions [1].
>
> The hadoop-common and hadoop-hdfs are okay, they compile without erros, but
> when I execute ant mvn-install to compile hadoop-mapreduce I get this error.
>
> compile-mapred-test:
> [javac] /home/lbd/hadoop/hadoop-ramiro/hadoop-mapreduce/build.xml:602:
> warning: 'includeantruntime' was not set, defaulting to
> build.sysclasspath=last; set to false for repeatable builds
> [javac] Compiling 179 source files to
> /home/lbd/hadoop/hadoop-ramiro/hadoop-mapreduce/build/test/mapred/classes
> [javac]
> /home/lbd/hadoop/hadoop-ramiro/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMRServerPorts.java:84:
> cannot find symbol
> [javac] symbol : variable NAME_NODE_HOST
> [javac] TestHDFSServerPorts.NAME_NODE_HOST + "0");
> [javac] ^
> [javac]
> /home/lbd/hadoop/hadoop-ramiro/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMRServerPorts.java:86:
> cannot find symbol
> [javac] symbol : variable NAME_NODE_HTTP_HOST
> [javac] location: class org.apache.hadoop.hdfs.TestHDFSServerPorts
> [javac] TestHDFSServerPorts.NAME_NODE_HTTP_HOST + "0");
> [javac] ^
> ...
>
> Is that a bug?
>
> This is my build.properties
>
> #this is essential
> resolvers=internal
> #you can increment this number as you see fit
> version=0.22.0-alpha-1
> project.version=${version}
> hadoop.version=${version}
> hadoop-core.version=${version}
> hadoop-hdfs.version=${version}
> hadoop-mapred.version=${version}
>
> Other question, Is the 0.22.0-alpha-1 the latest version?
>
> Thanks in advance,
>
> [1] https://github.com/apache/hadoop-mapreduce
>
> --
> Edson Ramiro Lucas Filho
> {skype, twitter, gtalk}: erlfilho
> http://www.inf.ufpr.br/erlf07/
>
Re: Java->native .so->seg fault->core dump file?
> My Java mappers use JNI to call native .so files compiled > from C++. In some cases, the task status ends with exit 139, > which generally indicates a seg-fault. I would like to see the > core-dump, but I can't seem to get it to work. No clue about 0.19, but does the owner of the process(es) in question have permission to write to the directory in question? We've seen a similar issue in which root or ops or somebody owns the HADOOP_HOME dir (which, IIRC, is where many of the processes get started), so neither mapred nor hdfs has permission to write anything there. Greg
Example for "Strings Comparison"
Dear all, I am looking for an example for a Mapreduce java implementation for strings comparison [pair-wise comparison]. If anybody have gone through a similar example before, could you please help pointing me to a code example for this? Thanks
Re: Losing Records with Block Compressed Sequence File
Make sure to close the output writer? I had similar problems in a different scenario and it turned out I was neglecting to close/flush my output. On 01/21/2011 01:04 PM, David Sinclair wrote: Hi, I am seeing an odd problem when writing block compressed sequence files. If I write 400,000 records into a sequence file w/o compression, all 400K end up in the file. If I write with block, regardless if it is bz2 or deflate, I start losing records. Not a ton, but a couple hundred. Here are the exact numbers bz2 399,734 deflate 399,770 none 400,000 Conf settings io.file.buffer.size - 4K, io.seqfile.compress.blocksize - 1MB anyone ever see this behavior? thanks dave
Losing Records with Block Compressed Sequence File
Hi, I am seeing an odd problem when writing block compressed sequence files. If I write 400,000 records into a sequence file w/o compression, all 400K end up in the file. If I write with block, regardless if it is bz2 or deflate, I start losing records. Not a ton, but a couple hundred. Here are the exact numbers bz2 399,734 deflate 399,770 none 400,000 Conf settings io.file.buffer.size - 4K, io.seqfile.compress.blocksize - 1MB anyone ever see this behavior? thanks dave
Java->native .so->seg fault->core dump file?
My Java mappers use JNI to call native .so files compiled from C++. In some
cases, the task status ends with exit 139, which generally indicates a
seg-fault. I would like to see the core-dump, but I can't seem to get it to
work.
I have this line in my driver setup (Yes, it's v.19):
conf.setBoolean("keep.failed.task.files", true);
and I have this in my .bashrc (which I believe should be propagated to the
slave nodes):
ulimit -c unlimited
and in my native code I call rlimit() and write the results, where I see:
RLIMIT_CORE: 18446744073709551615 18446744073709551615
which indicates the "unlimited" setting, but I can't find any core dump files
in the node's hadoop directories after the job runs.
Any ideas what I'm doing wrong?
Keith Wiley kwi...@keithwiley.com www.keithwiley.com
"It's a fine line between meticulous and obsessive-compulsive and a slippery
rope between obsessive-compulsive and debilitatingly slow."
-- Keith Wiley
Re: start anyways with missing blocks
yess! thanks, that was exactly what i wanted. -mike On Fri, Jan 21, 2011 at 3:16 PM, Brian Bockelman wrote: > Hi Mike, > > You want to take things out of safemode before you can make these changes. > > hadoop dfsadmin -safemode leave > > Then you can do the "hadoop fsck / -delete" > > Brian > > On Jan 21, 2011, at 2:12 PM, mike anderson wrote: > > > Also, here's the output of dfsadmin -report. What seems weird is that > it's > > not reporting any missing blocks. BTW, I tried doing fsck / -delete, but > it > > failed, complaining about the missing nodes. > > > > $ ../bin/hadoop dfsadmin -report > > Safe mode is ON > > Configured Capacity: 3915872829440 (3.56 TB) > > Present Capacity: 2913577631744 (2.65 TB) > > DFS Remaining: 1886228164608 (1.72 TB) > > DFS Used: 1027349467136 (956.79 GB) > > DFS Used%: 35.26% > > Under replicated blocks: 0 > > Blocks with corrupt replicas: 0 > > Missing blocks: 0 > > > > - > > Datanodes available: 9 (9 total, 0 dead) > > > > Name: 10.0.16.91:50010 > > Decommission Status : Normal > > Configured Capacity: 139438620672 (129.86 GB) > > DFS Used: 44507017216 (41.45 GB) > > Non DFS Used: 85782597632 (79.89 GB) > > DFS Remaining: 9149005824(8.52 GB) > > DFS Used%: 31.92% > > DFS Remaining%: 6.56% > > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > > > > Name: 10.0.16.165:50010 > > Decommission Status : Normal > > Configured Capacity: 472054276096 (439.63 GB) > > DFS Used: 139728683008 (130.13 GB) > > Non DFS Used: 90374217728 (84.17 GB) > > DFS Remaining: 241951375360(225.33 GB) > > DFS Used%: 29.6% > > DFS Remaining%: 51.25% > > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > > > > Name: 10.0.16.163:50010 > > Decommission Status : Normal > > Configured Capacity: 472054276096 (439.63 GB) > > DFS Used: 174687391744 (162.69 GB) > > Non DFS Used: 55780028416 (51.95 GB) > > DFS Remaining: 241586855936(225 GB) > > DFS Used%: 37.01% > > DFS Remaining%: 51.18% > > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > > > > Name: 10.0.16.164:50010 > > Decommission Status : Normal > > Configured Capacity: 472054276096 (439.63 GB) > > DFS Used: 95075942400 (88.55 GB) > > Non DFS Used: 182544318464 (170.01 GB) > > DFS Remaining: 194434015232(181.08 GB) > > DFS Used%: 20.14% > > DFS Remaining%: 41.19% > > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > > > > Name: 10.0.16.169:50010 > > Decommission Status : Normal > > Configured Capacity: 472054276096 (439.63 GB) > > DFS Used: 24576 (24 KB) > > Non DFS Used: 51301322752 (47.78 GB) > > DFS Remaining: 420752928768(391.86 GB) > > DFS Used%: 0% > > DFS Remaining%: 89.13% > > Last contact: Fri Jan 21 15:10:48 EST 2011 > > > > > > Name: 10.0.16.160:50010 > > Decommission Status : Normal > > Configured Capacity: 472054276096 (439.63 GB) > > DFS Used: 171275218944 (159.51 GB) > > Non DFS Used: 119652265984 (111.43 GB) > > DFS Remaining: 181126791168(168.69 GB) > > DFS Used%: 36.28% > > DFS Remaining%: 38.37% > > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > > > > Name: 10.0.16.161:50010 > > Decommission Status : Normal > > Configured Capacity: 472054276096 (439.63 GB) > > DFS Used: 131355377664 (122.33 GB) > > Non DFS Used: 174232702976 (162.27 GB) > > DFS Remaining: 166466195456(155.03 GB) > > DFS Used%: 27.83% > > DFS Remaining%: 35.26% > > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > > > > Name: 10.0.16.162:50010 > > Decommission Status : Normal > > Configured Capacity: 472054276096 (439.63 GB) > > DFS Used: 139831177216 (130.23 GB) > > Non DFS Used: 91403055104 (85.13 GB) > > DFS Remaining: 240820043776(224.28 GB) > > DFS Used%: 29.62% > > DFS Remaining%: 51.02% > > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > > > > Name: 10.0.16.167:50010 > > Decommission Status : Normal > > Configured Capacity: 472054276096 (439.63 GB) > > DFS Used: 130888634368 (121.9 GB) > > Non DFS Used: 151224688640 (140.84 GB) > > DFS Remaining: 189940953088(176.9 GB) > > DFS Used%: 27.73% > > DFS Remaining%: 40.24% > > Last contact: Fri Jan 21 15:10:46 EST 2011 > > > > > > On Fri, Jan 21, 2011 at 3:03 PM, mike anderson >wrote: > > > >> After a tragic cluster crash it looks like some blocks are missing. > >> > >> Total size: 343918527293 B (Total open files size: 67108864 B) > >> Total dirs: 5897 > >> Total files: 5574 (Files currently being written: 19) > >> Total blocks (validated): 9441 (avg. block size 36428188 B) (Total open > >> file blocks (not validated): 1) > >> > >> CORRUPT FILES: 319 > >> MISSING BLOCKS: 691 > >> MISSING SIZE: 32767071153 B > >> CORRUPT BLOCKS: 691 > >> > >> Minimally replicated blocks: 8750 (92.68086 %) > >> Over-replicated blocks: 0 (0.0 %) > >> Under-replicated blocks: 0 (0.0 %) > >> Mis-replicated blocks: 0 (0.0 %) > >> Default replication factor: 2 > >> Average block replication: 2.731914 > >> Corrupt blocks: 691 > >> Missing replicas: 0 (0.0 %) > >> Number of data-nodes: 9 > >> Number of racks: 1 > >>
Re: start anyways with missing blocks
Hi Mike, You want to take things out of safemode before you can make these changes. hadoop dfsadmin -safemode leave Then you can do the "hadoop fsck / -delete" Brian On Jan 21, 2011, at 2:12 PM, mike anderson wrote: > Also, here's the output of dfsadmin -report. What seems weird is that it's > not reporting any missing blocks. BTW, I tried doing fsck / -delete, but it > failed, complaining about the missing nodes. > > $ ../bin/hadoop dfsadmin -report > Safe mode is ON > Configured Capacity: 3915872829440 (3.56 TB) > Present Capacity: 2913577631744 (2.65 TB) > DFS Remaining: 1886228164608 (1.72 TB) > DFS Used: 1027349467136 (956.79 GB) > DFS Used%: 35.26% > Under replicated blocks: 0 > Blocks with corrupt replicas: 0 > Missing blocks: 0 > > - > Datanodes available: 9 (9 total, 0 dead) > > Name: 10.0.16.91:50010 > Decommission Status : Normal > Configured Capacity: 139438620672 (129.86 GB) > DFS Used: 44507017216 (41.45 GB) > Non DFS Used: 85782597632 (79.89 GB) > DFS Remaining: 9149005824(8.52 GB) > DFS Used%: 31.92% > DFS Remaining%: 6.56% > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > Name: 10.0.16.165:50010 > Decommission Status : Normal > Configured Capacity: 472054276096 (439.63 GB) > DFS Used: 139728683008 (130.13 GB) > Non DFS Used: 90374217728 (84.17 GB) > DFS Remaining: 241951375360(225.33 GB) > DFS Used%: 29.6% > DFS Remaining%: 51.25% > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > Name: 10.0.16.163:50010 > Decommission Status : Normal > Configured Capacity: 472054276096 (439.63 GB) > DFS Used: 174687391744 (162.69 GB) > Non DFS Used: 55780028416 (51.95 GB) > DFS Remaining: 241586855936(225 GB) > DFS Used%: 37.01% > DFS Remaining%: 51.18% > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > Name: 10.0.16.164:50010 > Decommission Status : Normal > Configured Capacity: 472054276096 (439.63 GB) > DFS Used: 95075942400 (88.55 GB) > Non DFS Used: 182544318464 (170.01 GB) > DFS Remaining: 194434015232(181.08 GB) > DFS Used%: 20.14% > DFS Remaining%: 41.19% > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > Name: 10.0.16.169:50010 > Decommission Status : Normal > Configured Capacity: 472054276096 (439.63 GB) > DFS Used: 24576 (24 KB) > Non DFS Used: 51301322752 (47.78 GB) > DFS Remaining: 420752928768(391.86 GB) > DFS Used%: 0% > DFS Remaining%: 89.13% > Last contact: Fri Jan 21 15:10:48 EST 2011 > > > Name: 10.0.16.160:50010 > Decommission Status : Normal > Configured Capacity: 472054276096 (439.63 GB) > DFS Used: 171275218944 (159.51 GB) > Non DFS Used: 119652265984 (111.43 GB) > DFS Remaining: 181126791168(168.69 GB) > DFS Used%: 36.28% > DFS Remaining%: 38.37% > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > Name: 10.0.16.161:50010 > Decommission Status : Normal > Configured Capacity: 472054276096 (439.63 GB) > DFS Used: 131355377664 (122.33 GB) > Non DFS Used: 174232702976 (162.27 GB) > DFS Remaining: 166466195456(155.03 GB) > DFS Used%: 27.83% > DFS Remaining%: 35.26% > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > Name: 10.0.16.162:50010 > Decommission Status : Normal > Configured Capacity: 472054276096 (439.63 GB) > DFS Used: 139831177216 (130.23 GB) > Non DFS Used: 91403055104 (85.13 GB) > DFS Remaining: 240820043776(224.28 GB) > DFS Used%: 29.62% > DFS Remaining%: 51.02% > Last contact: Fri Jan 21 15:10:47 EST 2011 > > > Name: 10.0.16.167:50010 > Decommission Status : Normal > Configured Capacity: 472054276096 (439.63 GB) > DFS Used: 130888634368 (121.9 GB) > Non DFS Used: 151224688640 (140.84 GB) > DFS Remaining: 189940953088(176.9 GB) > DFS Used%: 27.73% > DFS Remaining%: 40.24% > Last contact: Fri Jan 21 15:10:46 EST 2011 > > > On Fri, Jan 21, 2011 at 3:03 PM, mike anderson wrote: > >> After a tragic cluster crash it looks like some blocks are missing. >> >> Total size: 343918527293 B (Total open files size: 67108864 B) >> Total dirs: 5897 >> Total files: 5574 (Files currently being written: 19) >> Total blocks (validated): 9441 (avg. block size 36428188 B) (Total open >> file blocks (not validated): 1) >> >> CORRUPT FILES: 319 >> MISSING BLOCKS: 691 >> MISSING SIZE: 32767071153 B >> CORRUPT BLOCKS: 691 >> >> Minimally replicated blocks: 8750 (92.68086 %) >> Over-replicated blocks: 0 (0.0 %) >> Under-replicated blocks: 0 (0.0 %) >> Mis-replicated blocks: 0 (0.0 %) >> Default replication factor: 2 >> Average block replication: 2.731914 >> Corrupt blocks: 691 >> Missing replicas: 0 (0.0 %) >> Number of data-nodes: 9 >> Number of racks: 1 >> >> >> The filesystem under path '/' is CORRUPT >> >> >> >> I don't particularly care if I lose some of the data (it's just a cache >> store), instead of figuring out where the blocks went missing can I just >> forget about them and boot up with the blocks I have? >> >> -Mike >> smime.p7s Description: S/MIME cryptographic signature
Re: start anyways with missing blocks
Also, here's the output of dfsadmin -report. What seems weird is that it's not reporting any missing blocks. BTW, I tried doing fsck / -delete, but it failed, complaining about the missing nodes. $ ../bin/hadoop dfsadmin -report Safe mode is ON Configured Capacity: 3915872829440 (3.56 TB) Present Capacity: 2913577631744 (2.65 TB) DFS Remaining: 1886228164608 (1.72 TB) DFS Used: 1027349467136 (956.79 GB) DFS Used%: 35.26% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 - Datanodes available: 9 (9 total, 0 dead) Name: 10.0.16.91:50010 Decommission Status : Normal Configured Capacity: 139438620672 (129.86 GB) DFS Used: 44507017216 (41.45 GB) Non DFS Used: 85782597632 (79.89 GB) DFS Remaining: 9149005824(8.52 GB) DFS Used%: 31.92% DFS Remaining%: 6.56% Last contact: Fri Jan 21 15:10:47 EST 2011 Name: 10.0.16.165:50010 Decommission Status : Normal Configured Capacity: 472054276096 (439.63 GB) DFS Used: 139728683008 (130.13 GB) Non DFS Used: 90374217728 (84.17 GB) DFS Remaining: 241951375360(225.33 GB) DFS Used%: 29.6% DFS Remaining%: 51.25% Last contact: Fri Jan 21 15:10:47 EST 2011 Name: 10.0.16.163:50010 Decommission Status : Normal Configured Capacity: 472054276096 (439.63 GB) DFS Used: 174687391744 (162.69 GB) Non DFS Used: 55780028416 (51.95 GB) DFS Remaining: 241586855936(225 GB) DFS Used%: 37.01% DFS Remaining%: 51.18% Last contact: Fri Jan 21 15:10:47 EST 2011 Name: 10.0.16.164:50010 Decommission Status : Normal Configured Capacity: 472054276096 (439.63 GB) DFS Used: 95075942400 (88.55 GB) Non DFS Used: 182544318464 (170.01 GB) DFS Remaining: 194434015232(181.08 GB) DFS Used%: 20.14% DFS Remaining%: 41.19% Last contact: Fri Jan 21 15:10:47 EST 2011 Name: 10.0.16.169:50010 Decommission Status : Normal Configured Capacity: 472054276096 (439.63 GB) DFS Used: 24576 (24 KB) Non DFS Used: 51301322752 (47.78 GB) DFS Remaining: 420752928768(391.86 GB) DFS Used%: 0% DFS Remaining%: 89.13% Last contact: Fri Jan 21 15:10:48 EST 2011 Name: 10.0.16.160:50010 Decommission Status : Normal Configured Capacity: 472054276096 (439.63 GB) DFS Used: 171275218944 (159.51 GB) Non DFS Used: 119652265984 (111.43 GB) DFS Remaining: 181126791168(168.69 GB) DFS Used%: 36.28% DFS Remaining%: 38.37% Last contact: Fri Jan 21 15:10:47 EST 2011 Name: 10.0.16.161:50010 Decommission Status : Normal Configured Capacity: 472054276096 (439.63 GB) DFS Used: 131355377664 (122.33 GB) Non DFS Used: 174232702976 (162.27 GB) DFS Remaining: 166466195456(155.03 GB) DFS Used%: 27.83% DFS Remaining%: 35.26% Last contact: Fri Jan 21 15:10:47 EST 2011 Name: 10.0.16.162:50010 Decommission Status : Normal Configured Capacity: 472054276096 (439.63 GB) DFS Used: 139831177216 (130.23 GB) Non DFS Used: 91403055104 (85.13 GB) DFS Remaining: 240820043776(224.28 GB) DFS Used%: 29.62% DFS Remaining%: 51.02% Last contact: Fri Jan 21 15:10:47 EST 2011 Name: 10.0.16.167:50010 Decommission Status : Normal Configured Capacity: 472054276096 (439.63 GB) DFS Used: 130888634368 (121.9 GB) Non DFS Used: 151224688640 (140.84 GB) DFS Remaining: 189940953088(176.9 GB) DFS Used%: 27.73% DFS Remaining%: 40.24% Last contact: Fri Jan 21 15:10:46 EST 2011 On Fri, Jan 21, 2011 at 3:03 PM, mike anderson wrote: > After a tragic cluster crash it looks like some blocks are missing. > > Total size: 343918527293 B (Total open files size: 67108864 B) > Total dirs: 5897 > Total files: 5574 (Files currently being written: 19) > Total blocks (validated): 9441 (avg. block size 36428188 B) (Total open > file blocks (not validated): 1) > > CORRUPT FILES: 319 > MISSING BLOCKS: 691 > MISSING SIZE: 32767071153 B > CORRUPT BLOCKS: 691 > > Minimally replicated blocks: 8750 (92.68086 %) > Over-replicated blocks: 0 (0.0 %) > Under-replicated blocks: 0 (0.0 %) > Mis-replicated blocks: 0 (0.0 %) > Default replication factor: 2 > Average block replication: 2.731914 > Corrupt blocks: 691 > Missing replicas: 0 (0.0 %) > Number of data-nodes: 9 > Number of racks: 1 > > > The filesystem under path '/' is CORRUPT > > > > I don't particularly care if I lose some of the data (it's just a cache > store), instead of figuring out where the blocks went missing can I just > forget about them and boot up with the blocks I have? > > -Mike >
mapred.child.ulimit default value of null?
What is the interpretation of a default value for mapred.child.ulimit of null? Does that mean children are completely unlimited in their permission to use memory? If I want to insure maximum memory availability to children, should I leave this null (on a cluster where that is the default), or should I set it to some fairly large number, say 8 GBs (bearing in mind that the parameter's units are KB)? Is null better given my desire here? Also, what is a "child" in the context of this question (what does this parameter apply to)? For example, is a map task itself considered a "child" of the overall job or does this parameter not affect the map task itself but only child processes spawned *by* the map task? In particular, how would this parameter (or any other relevant parameters you can think of, say, mapred.child.java.opts) affect the availability of memory to compiled C++ code (.so files in the distributed cache) called from a map task via JNI? What I really want to do is insure that the native routines (C++ .so) called via JNI from a (Java) mapper have maximum memory availability/access on the node. Ostensibly, the native routines would be permitted to use the entire memory available on the machine. Thank you. Keith Wiley kwi...@keithwiley.com www.keithwiley.com "And what if we picked the wrong religion? Every week, we're just making God madder and madder!" -- Homer Simpson
error compiling hadoop-mapreduce
Hi all,
I'm compiling hadoop from git using these instructions [1].
The hadoop-common and hadoop-hdfs are okay, they compile without erros, but
when I execute ant mvn-install to compile hadoop-mapreduce I get this error.
compile-mapred-test:
[javac] /home/lbd/hadoop/hadoop-ramiro/hadoop-mapreduce/build.xml:602:
warning: 'includeantruntime' was not set, defaulting to
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 179 source files to
/home/lbd/hadoop/hadoop-ramiro/hadoop-mapreduce/build/test/mapred/classes
[javac]
/home/lbd/hadoop/hadoop-ramiro/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMRServerPorts.java:84:
cannot find symbol
[javac] symbol : variable NAME_NODE_HOST
[javac] TestHDFSServerPorts.NAME_NODE_HOST + "0");
[javac]^
[javac]
/home/lbd/hadoop/hadoop-ramiro/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestMRServerPorts.java:86:
cannot find symbol
[javac] symbol : variable NAME_NODE_HTTP_HOST
[javac] location: class org.apache.hadoop.hdfs.TestHDFSServerPorts
[javac] TestHDFSServerPorts.NAME_NODE_HTTP_HOST + "0");
[javac]^
...
Is that a bug?
This is my build.properties
#this is essential
resolvers=internal
#you can increment this number as you see fit
version=0.22.0-alpha-1
project.version=${version}
hadoop.version=${version}
hadoop-core.version=${version}
hadoop-hdfs.version=${version}
hadoop-mapred.version=${version}
Other question, Is the 0.22.0-alpha-1 the latest version?
Thanks in advance,
[1] https://github.com/apache/hadoop-mapreduce
--
Edson Ramiro Lucas Filho
{skype, twitter, gtalk}: erlfilho
http://www.inf.ufpr.br/erlf07/
Re: Hive rc location
On Fri, Jan 21, 2011 at 9:56 AM, abhatna...@vantage.com wrote: > > Where is this file located? > > Also does anyone has a sample > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Hive-rc-tp2296028p2302262.html > Sent from the Hadoop lucene-users mailing list archive at Nabble.com. > Please register for the hive user or dev lists for these type of questions. Hadoop user and common user is not the ideal place for this question. Hive will look for the file in two locations! env[HIVE_HOME]/bin/.hiverc, property(user.home)/.hiverc https://issues.apache.org/jira/browse/HIVE-1414 However this feature is currently only in trunk! So if you are running hive-0.6.0 You do not have hiverc support yet. Edward
Re: Hive rc location
Where is this file located? Also does anyone has a sample -- View this message in context: http://lucene.472066.n3.nabble.com/Hive-rc-tp2296028p2302262.html Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Hadoop Online Prototype, Continous Queries
Hi Guys, I've been looking into near-real time analytics solutions and HOP came as a good starting point for me as a Hadoop user already. However, I've done some research on analyzing how Continous Queries/Jobs work on HOP however I could not find a documentation for the code or even sample code that does it. Are there some code hiding from me in repositories that needs to be discovered? Can anyone here suggest a starting point and a brief introduction on what type of job specs need to be set to the HOP cluster so that one would achieve Continous Queries. Regards, Utku
Re: Why Hadoop is slow in Cloud
On 21/01/11 09:20, Evert Lammerts wrote: Even with performance hit, there are still benefits running Hadoop this way -as you only consume/pay for CPU time you use, if you are only running batch jobs, its lower cost than having a hadoop cluster that is under- used. -if your data is stored in the cloud infrastructure, then you need to data mine it in VMs, unless you want to take the time and money hit of moving it out, and have somewhere to store it. -if the infrastructure lets you, you can lock down the cluster so it is secure. Where a physical cluster is good is that it is a very low cost way of storing data, provided you can analyse it with Hadoop, and provided you can keep that cluster busy most of the time, either with Hadoop work or other scheduled work. If your cluster is idle for computation, you are still paying the capital and (reduced) electricity costs, so the cost of storage and what compute you do effectively increases. Agreed, but this has little to do with Hadoop as a middleware and more to do with the benefits of virtualized vs physical infrastructure. I agree that it is convenient to use HDFS as a DFS to keep your data local to your VMs, but you could choose other DFS's as well. We don't use HDFS, we bring up VMs close to where the data persists. http://www.slideshare.net/steve_l/high-availability-hadoop The major benefit of Hadoop is its data-locality principle, and this is what you give up when you move to the cloud. Regardless of whether you store your data within your VM or on a NAS, it *will* have to travel over a line. As soon as that happens you lose the benefit of data-locality and are left with MapReduce as a way for parallel computing. And in that case you could use less restrictive software, like maybe PBS. You could even install HOD on your virtual cluster, if you'd like the possibility of MapReduce. We don't suffer locality hits so much, but you do pay for the extra infrastructure costs of a more agile datacentre, and if you go to redundancy in hardware over replication, you have less places to run your code. Even on EC2, which doesn't let you tell it what datasets you want to play with for its VM placer to use in its decisions, once data is in the datanodes you do get locality Adarsh, there are probably results around of more generic benchmark tools (Phoronix, POV-Ray, ...) for I/O and CPU performance in a VM. Those should give you a better idea of the penalties of virtualization. (Our experience with a number of technologies on our OpenNebula cloud is, like Steve points out, that you mainly pay for disk I/O performance.) -would be interesting to see anything you can publish there... I think a decision to go with either cloud or physical infrastructure should be based on the frequency, intensity and types of computation you expect on the short term (that should include operations dealing with data), and the way you think these parameters will develop on a mid-long term. And then compare the prices of a physical cluster that meets those demands (make sure to include power and operations) and the investment you would otherwise need to make in Cloud. +1
Re: Why Hadoop is slow in Cloud
On 20/01/11 23:24, Marc Farnum Rendino wrote: On Wed, Jan 19, 2011 at 2:50 PM, Edward Capriolo wrote: As for virtualization,paravirtualization,emulation.(whatever ulization) Wow; that's a really big category. There are always a lot of variables, but the net result is always less. It may be 2% 10% or 15%, but it is always less. If it's less of something I don't care about, it's not a factor (for me). On the other hand, if I'm paying less and getting more of what I DO care about, I'd rather go with that. It's about the cost/benefit *ratio*. There's also perf vs storage. On a big cluster, you could add a second Nehalem CPU and maybe get 10-15% boost on throughput, or for the same capex and opex add 10% new servers, which at scale means many more TB of storage and the compute to go with it. The decision rests with the team and their problems.
RE: Why Hadoop is slow in Cloud
> Even with performance hit, there are still benefits running Hadoop this > way > -as you only consume/pay for CPU time you use, if you are only > running > batch jobs, its lower cost than having a hadoop cluster that is under- > used. > > -if your data is stored in the cloud infrastructure, then you need to > data mine it in VMs, unless you want to take the time and money hit of > moving it out, and have somewhere to store it. > > -if the infrastructure lets you, you can lock down the cluster so it is > secure. > > Where a physical cluster is good is that it is a very low cost way of > storing data, provided you can analyse it with Hadoop, and provided you > can keep that cluster busy most of the time, either with Hadoop work or > other scheduled work. If your cluster is idle for computation, you are > still paying the capital and (reduced) electricity costs, so the cost > of > storage and what compute you do effectively increases. Agreed, but this has little to do with Hadoop as a middleware and more to do with the benefits of virtualized vs physical infrastructure. I agree that it is convenient to use HDFS as a DFS to keep your data local to your VMs, but you could choose other DFS's as well. The major benefit of Hadoop is its data-locality principle, and this is what you give up when you move to the cloud. Regardless of whether you store your data within your VM or on a NAS, it *will* have to travel over a line. As soon as that happens you lose the benefit of data-locality and are left with MapReduce as a way for parallel computing. And in that case you could use less restrictive software, like maybe PBS. You could even install HOD on your virtual cluster, if you'd like the possibility of MapReduce. Adarsh, there are probably results around of more generic benchmark tools (Phoronix, POV-Ray, ...) for I/O and CPU performance in a VM. Those should give you a better idea of the penalties of virtualization. (Our experience with a number of technologies on our OpenNebula cloud is, like Steve points out, that you mainly pay for disk I/O performance.) I think a decision to go with either cloud or physical infrastructure should be based on the frequency, intensity and types of computation you expect on the short term (that should include operations dealing with data), and the way you think these parameters will develop on a mid-long term. And then compare the prices of a physical cluster that meets those demands (make sure to include power and operations) and the investment you would otherwise need to make in Cloud. smime.p7s Description: S/MIME cryptographic signature
