Re: tools for scrubbing HDFS data nodes?

[Raghu Angadi]

Owen O'Malley wrote:


On Jan 28, 2009, at 6:16 PM, Sriram Rao wrote:


By "scrub" I mean, have a tool that reads every block on a given data
node.  That way, I'd be able to find corrupted blocks proactively
rather than having an app read the file and find it.


The datanode already has a thread that checks the blocks periodically 
for exactly that purpose.


since Hadoop 0.16.0. scans all the blocks every 3 weeks (by default, 
interval can be changed).


Raghu.

RE: Is Hadoop Suitable for me?

[Simon]

But we are looking for an open source solution.

If I do decide to implement this for the office storage, what problems will
I run into?

-Original Message-
From: Dmitry Pushkarev [mailto:u...@stanford.edu] 
Sent: Thursday, 29 January 2009 5:15 PM
To: core-user@hadoop.apache.org
Cc: sim...@bigair.net.au
Subject: RE: Is Hadoop Suitable for me?

Definitely not, 

You should be looking at expandable Ethernet storage that can be extended by
connecting additional SAS arrays. (like dell powervault and similar things
from other companies)

600Mb is just 6 seconds over gigabit network...

---
Dmitry Pushkarev


-Original Message-
From: Simon [mailto:sim...@bigair.net.au] 
Sent: Wednesday, January 28, 2009 10:02 PM
To: core-user@hadoop.apache.org
Subject: Is Hadoop Suitable for me?

Hi Hadoop Users,


I am trying to build a storage system for the office of about 20-30 users
which will store everything.

>From normal everyday documents to computer configuration files to big files
(600mb) which are generated every hour.

 

Is Hadoop suitable for this kind of environment?

 

Regards,

Simon



No virus found in this incoming message.
Checked by AVG - http://www.avg.com 
Version: 8.0.176 / Virus Database: 270.10.15/1921 - Release Date: 1/28/2009
6:37 AM

RE: Is Hadoop Suitable for me?

[Dmitry Pushkarev]

Definitely not, 

You should be looking at expandable Ethernet storage that can be extended by
connecting additional SAS arrays. (like dell powervault and similar things
from other companies)

600Mb is just 6 seconds over gigabit network...

---
Dmitry Pushkarev


-Original Message-
From: Simon [mailto:sim...@bigair.net.au] 
Sent: Wednesday, January 28, 2009 10:02 PM
To: core-user@hadoop.apache.org
Subject: Is Hadoop Suitable for me?

Hi Hadoop Users,


I am trying to build a storage system for the office of about 20-30 users
which will store everything.

>From normal everyday documents to computer configuration files to big files
(600mb) which are generated every hour.

 

Is Hadoop suitable for this kind of environment?

 

Regards,

Simon

Is Hadoop Suitable for me?

[Simon]

Hi Hadoop Users,


I am trying to build a storage system for the office of about 20-30 users
which will store everything.

>From normal everyday documents to computer configuration files to big files
(600mb) which are generated every hour.

 

Is Hadoop suitable for this kind of environment?

 

Regards,

Simon

Re: tools for scrubbing HDFS data nodes?

[Sriram Rao]

The failover is fine; we are more interested in finding corrupt blocks
sooner rather than later.  Since there is the thread in the datanode,
that is good.

The replication factor is 3.

Sriram

On Wed, Jan 28, 2009 at 6:45 PM, Sagar Naik  wrote:
> In addition to datanode itself finding corrupted blocks (As Owen mention)
> if the  client finds a corrupted - block, it will go to other replica
>
> Whts your replication factor ?
>
> -Sagar
>
> Sriram Rao wrote:
>>
>> Does this read every block of every file from all replicas and verify
>> that the checksums are good?
>>
>> Sriram
>>
>> On Wed, Jan 28, 2009 at 6:20 PM, Sagar Naik  wrote:
>>
>>>
>>> Check out fsck
>>>
>>> bin/hadoop fsck  -files -location -blocks
>>>
>>> Sriram Rao wrote:
>>>

 By "scrub" I mean, have a tool that reads every block on a given data
 node.  That way, I'd be able to find corrupted blocks proactively
 rather than having an app read the file and find it.

 Sriram

 On Wed, Jan 28, 2009 at 5:57 PM, Aaron Kimball 
 wrote:


>
> By "scrub" do you mean delete the blocks from the node?
>
> Read your conf/hadoop-site.xml file to determine where dfs.data.dir
> points,
> then for each directory in that list, just rm the directory. If you
> want
> to
> ensure that your data is preserved with appropriate replication levels
> on
> the rest of your clutser, you should use Hadoop's DataNode Decommission
> feature to up-replicate the data before you blow a copy away.
>
> - Aaron
>
> On Wed, Jan 28, 2009 at 2:10 PM, Sriram Rao 
> wrote:
>
>
>
>>
>> Hi,
>>
>> Is there a tool that one could run on a datanode to scrub all the
>> blocks on that node?
>>
>> Sriram
>>
>>
>>
>

Cannot run program "chmod": error=12, Not enough space

[Andy Liu]

I'm running Hadoop 0.19.0 on Solaris (SunOS 5.10 on x86) and many jobs are
failing with this exception:

Error initializing attempt_200901281655_0004_m_25_0:
java.io.IOException: Cannot run program "chmod": error=12, Not enough space
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:338)
at 
org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540)
at 
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532)
at 
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:274)
at 
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:487)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:468)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:375)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:367)
at 
org.apache.hadoop.mapred.MapTask.localizeConfiguration(MapTask.java:107)
at 
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:1803)
at 
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:1884)
at 
org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:784)
at 
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:778)
at 
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1636)
at 
org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:102)
at 
org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1602)
Caused by: java.io.IOException: error=12, Not enough space
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:53)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 20 more

However, all the disks have plenty of disk space left (over 800 gigs).  Can
somebody point me in the right direction?

Thanks,
Andy

Re: tools for scrubbing HDFS data nodes?

[Sagar Naik]

In addition to datanode itself finding corrupted blocks (As Owen mention)
if the  client finds a corrupted - block, it will go to other replica

Whts your replication factor ?

-Sagar

Sriram Rao wrote:

Does this read every block of every file from all replicas and verify
that the checksums are good?

Sriram

On Wed, Jan 28, 2009 at 6:20 PM, Sagar Naik  wrote:
  

Check out fsck

bin/hadoop fsck  -files -location -blocks

Sriram Rao wrote:


By "scrub" I mean, have a tool that reads every block on a given data
node.  That way, I'd be able to find corrupted blocks proactively
rather than having an app read the file and find it.

Sriram

On Wed, Jan 28, 2009 at 5:57 PM, Aaron Kimball  wrote:

  

By "scrub" do you mean delete the blocks from the node?

Read your conf/hadoop-site.xml file to determine where dfs.data.dir
points,
then for each directory in that list, just rm the directory. If you want
to
ensure that your data is preserved with appropriate replication levels on
the rest of your clutser, you should use Hadoop's DataNode Decommission
feature to up-replicate the data before you blow a copy away.

- Aaron

On Wed, Jan 28, 2009 at 2:10 PM, Sriram Rao  wrote:




Hi,

Is there a tool that one could run on a datanode to scrub all the
blocks on that node?

Sriram


  

Re: tools for scrubbing HDFS data nodes?

[Owen O'Malley]

On Jan 28, 2009, at 6:16 PM, Sriram Rao wrote:


By "scrub" I mean, have a tool that reads every block on a given data
node.  That way, I'd be able to find corrupted blocks proactively
rather than having an app read the file and find it.


The datanode already has a thread that checks the blocks periodically  
for exactly that purpose.


-- Owen

Re: tools for scrubbing HDFS data nodes?

[Sriram Rao]

Does this read every block of every file from all replicas and verify
that the checksums are good?

Sriram

On Wed, Jan 28, 2009 at 6:20 PM, Sagar Naik  wrote:
> Check out fsck
>
> bin/hadoop fsck  -files -location -blocks
>
> Sriram Rao wrote:
>>
>> By "scrub" I mean, have a tool that reads every block on a given data
>> node.  That way, I'd be able to find corrupted blocks proactively
>> rather than having an app read the file and find it.
>>
>> Sriram
>>
>> On Wed, Jan 28, 2009 at 5:57 PM, Aaron Kimball  wrote:
>>
>>>
>>> By "scrub" do you mean delete the blocks from the node?
>>>
>>> Read your conf/hadoop-site.xml file to determine where dfs.data.dir
>>> points,
>>> then for each directory in that list, just rm the directory. If you want
>>> to
>>> ensure that your data is preserved with appropriate replication levels on
>>> the rest of your clutser, you should use Hadoop's DataNode Decommission
>>> feature to up-replicate the data before you blow a copy away.
>>>
>>> - Aaron
>>>
>>> On Wed, Jan 28, 2009 at 2:10 PM, Sriram Rao  wrote:
>>>
>>>

 Hi,

 Is there a tool that one could run on a datanode to scrub all the
 blocks on that node?

 Sriram


>

Re: [ANNOUNCE] Registration for ApacheCon Europe 2009 is now open!

[Christophe Bisciglia]

I wanted to provide two additional notes about my talk on this list.

First, you're really coming to see Aaron Kimball and Tom White - I'm
working on getting that fixed on the conference pages.

Second, "my" talk is actually a full day of intermediate/advanced
Hadoop training on Monday. It will be similar in style to what we
offer locally (http://www.cloudera.com/hadoop-training), but will be
specifically targeted at more advanced users (we will not be building
an inverted index, and I have heard rumors the star wars kid may be
involved). Here's the conference page:
http://eu.apachecon.com/c/aceu2009/sessions/230 - you can register
using the conference registration system (even if you don't attend the
conference itself).

If your boss won't fly you to Amsterdam for conference, calling it
"training" has been known to help ;-)

We'll keep the cluster up and running for the whole week, and provide
support / advice for training participants throughout.

Cheers,
Christophe

On Tue, Jan 27, 2009 at 8:35 AM, Owen O'Malley  wrote:
> All,
>   I'm broadcasting this to all of the Hadoop dev and users lists, however,
> in the future I'll only send cross-subproject announcements to
> gene...@hadoop.apache.org. Please subscribe over there too! It is very low
> traffic.
>  Anyways, ApacheCon Europe is coming up in March. There are a range of
> Hadoop talks being given:
>
> Introduction to Hadoop by Owen O'Malley
> Hadoop Map/Reduce: Tuning and Debugging by Arun Murthy
> Pig - Making Hadoop Easy by Olga Natkovich
> Running Hadoop in the Cloud by Tom White
> Architectures for the Cloud by Steve Loughran
> Configuring Hadoop for Grid Services by Allen Wittenauer
> Dynamic Hadoop Clusters by Steve Loughran
> HBasics: An Introduction to Hadoop's Bid Data Database by Michael Stack
> Hadoop Tools and Tricks for Data Pipelines by Christophe Bisciglia
> Introducing Mahout: Apache Machine Learning by Grant Ingersoll
>
> -- Owen
>
> Begin forwarded message:
>
>> From: Shane Curcuru 
>> Date: January 27, 2009 6:15:25 AM PST
>> Subject: [ANN] Registration for ApacheCon Europe 2009 is now open!
>>
>> PMC moderators - please forward the below to any appropriate dev@ or
>> users@ lists so your larger community can hear about ApacheCon Europe.
>> Remember, ACEU09 has scheduled sessions spanning the breadth of the ASF's
>> projects, subprojects, and podlings, including at least: ActiveMQ,
>> SerivceMix, CXF, Axis2, Hadoop, Felix, Sling, Maven, Struts, Roller,
>> Shindig, Geronimo, Lucene, Solr, BSF, Mina, Directory, Tomcat, httpd,
>> Mahout, Bayeux, CouchDB, AntUnit, Jackrabbit, Archiva, Wicket, POI, Pig,
>> Synapse, Droids, Continuum.
>>
>>
>> ApacheCon EU 2009 registration is now open!
>> 23-27 March -- Mövenpick Hotel, Amsterdam, Netherlands
>> http://www.eu.apachecon.com/
>> 
>>
>> Registration for ApacheCon Europe 2009 is now open - act before early
>> bird prices expire 6 February.  Remember to book a room at the Mövenpick
>> and use the Registration Code: Special package attendees for the
>> conference registration, and get 150 Euros off your full conference
>> registration.
>>
>> Lower Costs - Thanks to new VAT tax laws, our prices this year are 19%
>> lower than last year in Europe!  We've also negotiated a Mövenpick rate
>> of a maximum of 155 Euros per night for attendees in our room block.
>>
>> Quick Links:
>>
>>  http://xrl.us/aceu09sp  See the schedule
>>  http://xrl.us/aceu09hp  Get your hotel room
>>  http://xrl.us/aceu09rp  Register for the conference
>>
>> Other important notes:
>>
>> - Geeks for Geeks is a new mini-track where we can feature advanced
>> technical content from project committers.  And our Hackathon on Monday
>> and Tuesday is open to all attendees - be sure to check it off in your
>> registration.
>>
>> - The Call for Papers for ApacheCon US 2009, held 2-6 November
>> 2009 in Oakland, CA, is open through 28 February, so get your
>> submissions in now.  This ApacheCon will feature special events with
>> some of the ASF's original founders in celebration of the 10th
>> anniversary of The Apache Software Foundation.
>>
>>  http://www.us.apachecon.com/c/acus2009/
>>
>> - Interested in sponsoring the ApacheCon conferences?  There are plenty
>> of sponsor packages available - please contact Delia Frees at
>> de...@apachecon.com for further information.
>>
>> ==
>> ApacheCon EU 2008: A week of Open Source at it's best!
>>
>> Hackathon - open to all! | Geeks for Geeks | Lunchtime Sessions
>> In-Depth Trainings | Multi-Track Sessions | BOFs | Business Panel
>> Lightning Talks | Receptions | Fast Feather Track | Expo... and more!
>>
>> - Shane Curcuru, on behalf of
>>  Noirin Shirley, Conference Lead,
>>  and the whole ApacheCon Europe 2009 Team
>>  http://www.eu.apachecon.com/  23-27 March -- Amsterdam, Netherlands
>>
>>
>
>

Re: tools for scrubbing HDFS data nodes?

[Sagar Naik]

Check out fsck

bin/hadoop fsck  -files -location -blocks

Sriram Rao wrote:

By "scrub" I mean, have a tool that reads every block on a given data
node.  That way, I'd be able to find corrupted blocks proactively
rather than having an app read the file and find it.

Sriram

On Wed, Jan 28, 2009 at 5:57 PM, Aaron Kimball  wrote:
  

By "scrub" do you mean delete the blocks from the node?

Read your conf/hadoop-site.xml file to determine where dfs.data.dir points,
then for each directory in that list, just rm the directory. If you want to
ensure that your data is preserved with appropriate replication levels on
the rest of your clutser, you should use Hadoop's DataNode Decommission
feature to up-replicate the data before you blow a copy away.

- Aaron

On Wed, Jan 28, 2009 at 2:10 PM, Sriram Rao  wrote:



Hi,

Is there a tool that one could run on a datanode to scrub all the
blocks on that node?

Sriram

  

Re: tools for scrubbing HDFS data nodes?

[Sriram Rao]

By "scrub" I mean, have a tool that reads every block on a given data
node.  That way, I'd be able to find corrupted blocks proactively
rather than having an app read the file and find it.

Sriram

On Wed, Jan 28, 2009 at 5:57 PM, Aaron Kimball  wrote:
> By "scrub" do you mean delete the blocks from the node?
>
> Read your conf/hadoop-site.xml file to determine where dfs.data.dir points,
> then for each directory in that list, just rm the directory. If you want to
> ensure that your data is preserved with appropriate replication levels on
> the rest of your clutser, you should use Hadoop's DataNode Decommission
> feature to up-replicate the data before you blow a copy away.
>
> - Aaron
>
> On Wed, Jan 28, 2009 at 2:10 PM, Sriram Rao  wrote:
>
>> Hi,
>>
>> Is there a tool that one could run on a datanode to scrub all the
>> blocks on that node?
>>
>> Sriram
>>
>

Re: tools for scrubbing HDFS data nodes?

[Aaron Kimball]

By "scrub" do you mean delete the blocks from the node?

Read your conf/hadoop-site.xml file to determine where dfs.data.dir points,
then for each directory in that list, just rm the directory. If you want to
ensure that your data is preserved with appropriate replication levels on
the rest of your clutser, you should use Hadoop's DataNode Decommission
feature to up-replicate the data before you blow a copy away.

- Aaron

On Wed, Jan 28, 2009 at 2:10 PM, Sriram Rao  wrote:

> Hi,
>
> Is there a tool that one could run on a datanode to scrub all the
> blocks on that node?
>
> Sriram
>

Re: sudden instability in 0.18.2

[Aaron Kimball]

Wow. How many subdirectories were there? how many jobs do you run a day?

- Aaron

On Wed, Jan 28, 2009 at 12:13 PM, David J. O'Dell wrote:

> It was failing on all the nodes both new and old.
> The problem was there were too many subdirectories under
> $HADOOP_HOME/logs/userlogs
> The fix was just to delete the subdirs and change this setting from 24
> hours(the default) to 2 hours.
> mapred.userlog.retain.hours
>
> Would have been nice if there was an error message that pointed to this.
>
>
> Aaron Kimball wrote:
> > Hi David,
> >
> > If your tasks are failing on only the new nodes, it's likely that you're
> > missing a library or something on those machines. See this Hadoop
> tutorial
> > http://public.yahoo.com/gogate/hadoop-tutorial/html/module5.html about
> > "distributing debug scripts." These will allow you to capture stdout/err
> and
> > the syslog from tasks that fail.
> >
> > - Aaron
> >
> > On Wed, Jan 28, 2009 at 9:40 AM, Sagar Naik 
> wrote:
> >
> >
> >> Pl check which nodes have these failures.
> >>
> >> I guess the new tasktrackers/machines  are not configured correctly.
> >> As a result, the map-task will die and the remaining map-tasks will be
> >> sucked onto these machines
> >>
> >>
> >> -Sagar
> >>
> >>
> >> David J. O'Dell wrote:
> >>
> >>
> >>> We've been running 0.18.2 for over a month on an 8 node cluster.
> >>> Last week we added 4 more nodes to the cluster and have experienced 2
> >>> failures to the tasktrackers since then.
> >>> The namenodes are running fine but all jobs submitted will die when
> >>> submitted with this error on the tasktrackers.
> >>>
> >>> 2009-01-28 08:07:55,556 INFO org.apache.hadoop.mapred.TaskTracker:
> >>> LaunchTaskAction: attempt_200901280756_0012_m_74_2
> >>> 2009-01-28 08:07:55,682 WARN org.apache.hadoop.mapred.TaskRunner:
> >>> attempt_200901280756_0012_m_74_2 Child Error
> >>> java.io.IOException: Task process exit with nonzero status of 1.
> >>>at
> >>> org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:462)
> >>>at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:403)
> >>>
> >>> I tried running the tasktrackers in debug mode but the entries above
> are
> >>> all that show up in the logs.
> >>> As of now my cluster is down.
> >>>
> >>>
> >>>
> >>>
>
> --
> David O'Dell
> Director, Operations
> e: dod...@videoegg.com
> t:  (415) 738-5152
> 180 Townsend St., Third Floor
> San Francisco, CA 94107
>
>

Re: Hadoop+s3 & fuse-dfs

[Craig Macdonald]

Hi Roopa,

Glad it worked :-)

Please file JIRA issues against the fuse-dfs / libhdfs components that 
would have made it easier to mount the S3 filesystem.


Craig

Roopa Sudheendra wrote:
Thanks, Yes a setup with fuse-dfs and hdfs works fine.I think the 
mount point was bad for whatever reason and was failing with that 
error .I created another mount point for mounting which resolved  the 
transport end point error.


Also i had -d option on my command..:)


Roopa


On Jan 28, 2009, at 6:35 PM, Craig Macdonald wrote:


Hi Roopa,

Firstly, can you get the fuse-dfs working for an instance HDFS?
There is also a debug mode for fuse: enable this by adding -d on the 
command line.


C

Roopa Sudheendra wrote:

Hey Craig,
I tried the way u suggested..but i get this transport endpoint not 
connected. Can i see the logs anywhere? I dont see anything in 
/var/log/messages either
looks like it tries to create the file system in hdfs.c but not sure 
where it fails.

I have the hadoop home set so i believe it gets the config info.

any idea?

Thanks,
Roopa
On Jan 28, 2009, at 1:59 PM, Craig Macdonald wrote:


In theory, yes.
On inspection of libhdfs, which underlies fuse-dfs, I note that:

* libhdfs takes a host and port number as input when connecting, 
but not a scheme (hdfs etc). The easiest option would be to set the 
S3 as your default file system in your hadoop-site.xml, then use 
the host of "default". That should get libhdfs to use the S3 file 
system. i.e. set fuse-dfs to mount dfs://default:0/ and all should 
work as planned.


* libhdfs also casts the FileSystem to a DistributedFileSystem for 
the df command. This would fail in your case. This issue is 
currently being worked on - see HADOOP-4368

https://issues.apache.org/jira/browse/HADOOP-4368.

C


Roopa Sudheendra wrote:

Thanks for the response craig.
I looked at fuse-dfs c code and looks like it does not like 
anything other than "dfs:// " so with the fact that hadoop can 
connect to S3 file system ..allowing s3 scheme should solve my 
problem?


Roopa

On Jan 28, 2009, at 1:03 PM, Craig Macdonald wrote:


Hi Roopa,

I cant comment on the S3 specifics. However, fuse-dfs is based on 
a C interface called libhdfs which allows C programs (such as 
fuse-dfs) to connect to the Hadoop file system Java API. This 
being the case, fuse-dfs should (theoretically) be able to 
connect to any file system that Hadoop can. Your mileage may 
vary, but if you find issues, please do report them through the 
normal channels.


Craig


Roopa Sudheendra wrote:
I am experimenting with Hadoop backed by Amazon s3 filesystem as 
one of our backup storage solution. Just the hadoop and s3(block 
based since it overcomes the 5gb limit) so far seems to be fine.
My problem is that i want to mount this filesystem using 
fuse-dfs ( since i don't have to worry about how the file is 
written on the system ) . Since the namenode does not get 
started with s3 backed hadoop system how can i connect fuse-dfs 
to this setup.


Appreciate your help.
Thanks,
Roopa

Re: Hadoop+s3 & fuse-dfs

[Roopa Sudheendra]

Thanks, Yes a setup with fuse-dfs and hdfs works fine.I think the  
mount point was bad for whatever reason and was failing with that  
error .I created another mount point for mounting which resolved  the  
transport end point error.


Also i had -d option on my command..:)


Roopa


On Jan 28, 2009, at 6:35 PM, Craig Macdonald wrote:


Hi Roopa,

Firstly, can you get the fuse-dfs working for an instance HDFS?
There is also a debug mode for fuse: enable this by adding -d on the  
command line.


C

Roopa Sudheendra wrote:

Hey Craig,
I tried the way u suggested..but i get this transport endpoint not  
connected. Can i see the logs anywhere? I dont see anything in /var/ 
log/messages either
looks like it tries to create the file system in hdfs.c but not  
sure where it fails.

I have the hadoop home set so i believe it gets the config info.

any idea?

Thanks,
Roopa
On Jan 28, 2009, at 1:59 PM, Craig Macdonald wrote:


In theory, yes.
On inspection of libhdfs, which underlies fuse-dfs, I note that:

* libhdfs takes a host and port number as input when connecting,  
but not a scheme (hdfs etc). The easiest option would be to set  
the S3 as your default file system in your hadoop-site.xml, then  
use the host of "default". That should get libhdfs to use the S3  
file system. i.e. set fuse-dfs to mount dfs://default:0/ and all  
should work as planned.


* libhdfs also casts the FileSystem to a DistributedFileSystem for  
the df command. This would fail in your case. This issue is  
currently being worked on - see HADOOP-4368

https://issues.apache.org/jira/browse/HADOOP-4368.

C


Roopa Sudheendra wrote:

Thanks for the response craig.
I looked at fuse-dfs c code and looks like it does not like  
anything other than "dfs:// " so with the fact that hadoop can  
connect to S3 file system ..allowing s3 scheme should solve my  
problem?


Roopa

On Jan 28, 2009, at 1:03 PM, Craig Macdonald wrote:


Hi Roopa,

I cant comment on the S3 specifics. However, fuse-dfs is based  
on a C interface called libhdfs which allows C programs (such as  
fuse-dfs) to connect to the Hadoop file system Java API. This  
being the case, fuse-dfs should (theoretically) be able to  
connect to any file system that Hadoop can. Your mileage may  
vary, but if you find issues, please do report them through the  
normal channels.


Craig


Roopa Sudheendra wrote:
I am experimenting with Hadoop backed by Amazon s3 filesystem  
as one of our backup storage solution. Just the hadoop and  
s3(block based since it overcomes the 5gb limit) so far seems  
to be fine.
My problem is that i want to mount this filesystem using fuse- 
dfs ( since i don't have to worry about how the file is written  
on the system ) . Since the namenode does not get started with  
s3 backed hadoop system how can i connect fuse-dfs to this setup.


Appreciate your help.
Thanks,
Roopa

Re: Hadoop+s3 & fuse-dfs

[Craig Macdonald]

Hi Roopa,

Firstly, can you get the fuse-dfs working for an instance HDFS?
There is also a debug mode for fuse: enable this by adding -d on the 
command line.


C

Roopa Sudheendra wrote:

Hey Craig,
 I tried the way u suggested..but i get this transport endpoint not 
connected. Can i see the logs anywhere? I dont see anything in 
/var/log/messages either
 looks like it tries to create the file system in hdfs.c but not sure 
where it fails.

I have the hadoop home set so i believe it gets the config info.

any idea?

Thanks,
Roopa
On Jan 28, 2009, at 1:59 PM, Craig Macdonald wrote:


In theory, yes.
On inspection of libhdfs, which underlies fuse-dfs, I note that:

* libhdfs takes a host and port number as input when connecting, but 
not a scheme (hdfs etc). The easiest option would be to set the S3 as 
your default file system in your hadoop-site.xml, then use the host 
of "default". That should get libhdfs to use the S3 file system. i.e. 
set fuse-dfs to mount dfs://default:0/ and all should work as planned.


* libhdfs also casts the FileSystem to a DistributedFileSystem for 
the df command. This would fail in your case. This issue is currently 
being worked on - see HADOOP-4368

https://issues.apache.org/jira/browse/HADOOP-4368.

C


Roopa Sudheendra wrote:

Thanks for the response craig.
I looked at fuse-dfs c code and looks like it does not like anything 
other than "dfs:// " so with the fact that hadoop can connect to S3 
file system ..allowing s3 scheme should solve my problem?


Roopa

On Jan 28, 2009, at 1:03 PM, Craig Macdonald wrote:


Hi Roopa,

I cant comment on the S3 specifics. However, fuse-dfs is based on a 
C interface called libhdfs which allows C programs (such as 
fuse-dfs) to connect to the Hadoop file system Java API. This being 
the case, fuse-dfs should (theoretically) be able to connect to any 
file system that Hadoop can. Your mileage may vary, but if you find 
issues, please do report them through the normal channels.


Craig


Roopa Sudheendra wrote:
I am experimenting with Hadoop backed by Amazon s3 filesystem as 
one of our backup storage solution. Just the hadoop and s3(block 
based since it overcomes the 5gb limit) so far seems to be fine.
My problem is that i want to mount this filesystem using fuse-dfs 
( since i don't have to worry about how the file is written on the 
system ) . Since the namenode does not get started with s3 backed 
hadoop system how can i connect fuse-dfs to this setup.


Appreciate your help.
Thanks,
Roopa

Re: Hadoop+s3 & fuse-dfs

[Roopa Sudheendra]

Hey Craig,
 I tried the way u suggested..but i get this transport endpoint not  
connected. Can i see the logs anywhere? I dont see anything in /var/ 
log/messages either
 looks like it tries to create the file system in hdfs.c but not sure  
where it fails.

I have the hadoop home set so i believe it gets the config info.

any idea?

Thanks,
Roopa
On Jan 28, 2009, at 1:59 PM, Craig Macdonald wrote:


In theory, yes.
On inspection of libhdfs, which underlies fuse-dfs, I note that:

* libhdfs takes a host and port number as input when connecting, but  
not a scheme (hdfs etc). The easiest option would be to set the S3  
as your default file system in your hadoop-site.xml, then use the  
host of "default". That should get libhdfs to use the S3 file  
system. i.e. set fuse-dfs to mount dfs://default:0/ and all should  
work as planned.


* libhdfs also casts the FileSystem to a DistributedFileSystem for  
the df command. This would fail in your case. This issue is  
currently being worked on - see HADOOP-4368

https://issues.apache.org/jira/browse/HADOOP-4368.

C


Roopa Sudheendra wrote:

Thanks for the response craig.
I looked at fuse-dfs c code and looks like it does not like  
anything other than "dfs:// " so with the fact that hadoop can  
connect to S3 file system ..allowing s3 scheme should solve my  
problem?


Roopa

On Jan 28, 2009, at 1:03 PM, Craig Macdonald wrote:


Hi Roopa,

I cant comment on the S3 specifics. However, fuse-dfs is based on  
a C interface called libhdfs which allows C programs (such as fuse- 
dfs) to connect to the Hadoop file system Java API. This being the  
case, fuse-dfs should (theoretically) be able to connect to any  
file system that Hadoop can. Your mileage may vary, but if you  
find issues, please do report them through the normal channels.


Craig


Roopa Sudheendra wrote:
I am experimenting with Hadoop backed by Amazon s3 filesystem as  
one of our backup storage solution. Just the hadoop and s3(block  
based since it overcomes the 5gb limit) so far seems to be fine.
My problem is that i want to mount this filesystem using fuse-dfs  
( since i don't have to worry about how the file is written on  
the system ) . Since the namenode does not get started with s3  
backed hadoop system how can i connect fuse-dfs to this setup.


Appreciate your help.
Thanks,
Roopa

tools for scrubbing HDFS data nodes?

[Sriram Rao]

Hi,

Is there a tool that one could run on a datanode to scrub all the
blocks on that node?

Sriram

Re: sudden instability in 0.18.2

[David J. O'Dell]

It was failing on all the nodes both new and old.
The problem was there were too many subdirectories under
$HADOOP_HOME/logs/userlogs
The fix was just to delete the subdirs and change this setting from 24
hours(the default) to 2 hours.
mapred.userlog.retain.hours

Would have been nice if there was an error message that pointed to this.


Aaron Kimball wrote:
> Hi David,
>
> If your tasks are failing on only the new nodes, it's likely that you're
> missing a library or something on those machines. See this Hadoop tutorial
> http://public.yahoo.com/gogate/hadoop-tutorial/html/module5.html about
> "distributing debug scripts." These will allow you to capture stdout/err and
> the syslog from tasks that fail.
>
> - Aaron
>
> On Wed, Jan 28, 2009 at 9:40 AM, Sagar Naik  wrote:
>
>   
>> Pl check which nodes have these failures.
>>
>> I guess the new tasktrackers/machines  are not configured correctly.
>> As a result, the map-task will die and the remaining map-tasks will be
>> sucked onto these machines
>>
>>
>> -Sagar
>>
>>
>> David J. O'Dell wrote:
>>
>> 
>>> We've been running 0.18.2 for over a month on an 8 node cluster.
>>> Last week we added 4 more nodes to the cluster and have experienced 2
>>> failures to the tasktrackers since then.
>>> The namenodes are running fine but all jobs submitted will die when
>>> submitted with this error on the tasktrackers.
>>>
>>> 2009-01-28 08:07:55,556 INFO org.apache.hadoop.mapred.TaskTracker:
>>> LaunchTaskAction: attempt_200901280756_0012_m_74_2
>>> 2009-01-28 08:07:55,682 WARN org.apache.hadoop.mapred.TaskRunner:
>>> attempt_200901280756_0012_m_74_2 Child Error
>>> java.io.IOException: Task process exit with nonzero status of 1.
>>>at
>>> org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:462)
>>>at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:403)
>>>
>>> I tried running the tasktrackers in debug mode but the entries above are
>>> all that show up in the logs.
>>> As of now my cluster is down.
>>>
>>>
>>>
>>>   

-- 
David O'Dell
Director, Operations
e: dod...@videoegg.com
t:  (415) 738-5152
180 Townsend St., Third Floor
San Francisco, CA 94107 

Re: Hadoop+s3 & fuse-dfs

[Craig Macdonald]

In theory, yes.
On inspection of libhdfs, which underlies fuse-dfs, I note that:

* libhdfs takes a host and port number as input when connecting, but 
not a scheme (hdfs etc). The easiest option would be to set the S3 as 
your default file system in your hadoop-site.xml, then use the host of 
"default". That should get libhdfs to use the S3 file system. i.e. set 
fuse-dfs to mount dfs://default:0/ and all should work as planned.


* libhdfs also casts the FileSystem to a DistributedFileSystem for the 
df command. This would fail in your case. This issue is currently being 
worked on - see HADOOP-4368

https://issues.apache.org/jira/browse/HADOOP-4368.

C


Roopa Sudheendra wrote:

Thanks for the response craig.
I looked at fuse-dfs c code and looks like it does not like anything 
other than "dfs:// " so with the fact that hadoop can connect to S3 
file system ..allowing s3 scheme should solve my problem?


Roopa

On Jan 28, 2009, at 1:03 PM, Craig Macdonald wrote:


Hi Roopa,

I cant comment on the S3 specifics. However, fuse-dfs is based on a C 
interface called libhdfs which allows C programs (such as fuse-dfs) 
to connect to the Hadoop file system Java API. This being the case, 
fuse-dfs should (theoretically) be able to connect to any file system 
that Hadoop can. Your mileage may vary, but if you find issues, 
please do report them through the normal channels.


Craig


Roopa Sudheendra wrote:
I am experimenting with Hadoop backed by Amazon s3 filesystem as one 
of our backup storage solution. Just the hadoop and s3(block based 
since it overcomes the 5gb limit) so far seems to be fine.
My problem is that i want to mount this filesystem using fuse-dfs ( 
since i don't have to worry about how the file is written on the 
system ) . Since the namenode does not get started with s3 backed 
hadoop system how can i connect fuse-dfs to this setup.


Appreciate your help.
Thanks,
Roopa

Re: Hadoop+s3 & fuse-dfs

[Roopa Sudheendra]

Thanks for the response craig.
I looked at fuse-dfs c code and looks like it does not like anything  
other than "dfs:// " so with the fact that hadoop can connect to S3  
file system ..allowing s3 scheme should solve my problem?


Roopa

On Jan 28, 2009, at 1:03 PM, Craig Macdonald wrote:


Hi Roopa,

I cant comment on the S3 specifics. However, fuse-dfs is based on a  
C interface called libhdfs which allows C programs (such as fuse- 
dfs) to connect to the Hadoop file system Java API. This being the  
case, fuse-dfs should (theoretically) be able to connect to any file  
system that Hadoop can. Your mileage may vary, but if you find  
issues, please do report them through the normal channels.


Craig


Roopa Sudheendra wrote:
I am experimenting with Hadoop backed by Amazon s3 filesystem as  
one of our backup storage solution. Just the hadoop and s3(block  
based since it overcomes the 5gb limit) so far seems to be fine.
My problem is that i want to mount this filesystem using fuse-dfs  
( since i don't have to worry about how the file is written on the  
system ) . Since the namenode does not get started with s3 backed  
hadoop system how can i connect fuse-dfs to this setup.


Appreciate your help.
Thanks,
Roopa

Re: sudden instability in 0.18.2

[Aaron Kimball]

Hi David,

If your tasks are failing on only the new nodes, it's likely that you're
missing a library or something on those machines. See this Hadoop tutorial
http://public.yahoo.com/gogate/hadoop-tutorial/html/module5.html about
"distributing debug scripts." These will allow you to capture stdout/err and
the syslog from tasks that fail.

- Aaron

On Wed, Jan 28, 2009 at 9:40 AM, Sagar Naik  wrote:

> Pl check which nodes have these failures.
>
> I guess the new tasktrackers/machines  are not configured correctly.
> As a result, the map-task will die and the remaining map-tasks will be
> sucked onto these machines
>
>
> -Sagar
>
>
> David J. O'Dell wrote:
>
>> We've been running 0.18.2 for over a month on an 8 node cluster.
>> Last week we added 4 more nodes to the cluster and have experienced 2
>> failures to the tasktrackers since then.
>> The namenodes are running fine but all jobs submitted will die when
>> submitted with this error on the tasktrackers.
>>
>> 2009-01-28 08:07:55,556 INFO org.apache.hadoop.mapred.TaskTracker:
>> LaunchTaskAction: attempt_200901280756_0012_m_74_2
>> 2009-01-28 08:07:55,682 WARN org.apache.hadoop.mapred.TaskRunner:
>> attempt_200901280756_0012_m_74_2 Child Error
>> java.io.IOException: Task process exit with nonzero status of 1.
>>at
>> org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:462)
>>at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:403)
>>
>> I tried running the tasktrackers in debug mode but the entries above are
>> all that show up in the logs.
>> As of now my cluster is down.
>>
>>
>>
>

Re: Hadoop+s3 & fuse-dfs

[Craig Macdonald]

Hi Roopa,

I cant comment on the S3 specifics. However, fuse-dfs is based on a C 
interface called libhdfs which allows C programs (such as fuse-dfs) to 
connect to the Hadoop file system Java API. This being the case, 
fuse-dfs should (theoretically) be able to connect to any file system 
that Hadoop can. Your mileage may vary, but if you find issues, please 
do report them through the normal channels.


Craig


Roopa Sudheendra wrote:
I am experimenting with Hadoop backed by Amazon s3 filesystem as one 
of our backup storage solution. Just the hadoop and s3(block based 
since it overcomes the 5gb limit) so far seems to be fine.
My problem is that i want to mount this filesystem using fuse-dfs ( 
since i don't have to worry about how the file is written on the 
system ) . Since the namenode does not get started with s3 backed 
hadoop system how can i connect fuse-dfs to this setup.


Appreciate your help.
Thanks,
Roopa

Hadoop+s3 & fuse-dfs

[Roopa Sudheendra]

I am experimenting with Hadoop backed by Amazon s3 filesystem as one  
of our backup storage solution. Just the hadoop and s3(block based  
since it overcomes the 5gb limit) so far seems to be fine.
My problem is that i want to mount this filesystem using fuse-dfs  
( since i don't have to worry about how the file is written on the  
system ) . Since the namenode does not get started with s3 backed  
hadoop system how can i connect fuse-dfs to this setup.


Appreciate your help.
Thanks,
Roopa

Re: sudden instability in 0.18.2

[Sagar Naik]

Pl check which nodes have these failures.

I guess the new tasktrackers/machines  are not configured correctly.
As a result, the map-task will die and the remaining map-tasks will be 
sucked onto these machines



-Sagar

David J. O'Dell wrote:

We've been running 0.18.2 for over a month on an 8 node cluster.
Last week we added 4 more nodes to the cluster and have experienced 2
failures to the tasktrackers since then.
The namenodes are running fine but all jobs submitted will die when
submitted with this error on the tasktrackers.

2009-01-28 08:07:55,556 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction: attempt_200901280756_0012_m_74_2
2009-01-28 08:07:55,682 WARN org.apache.hadoop.mapred.TaskRunner:
attempt_200901280756_0012_m_74_2 Child Error
java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:462)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:403)

I tried running the tasktrackers in debug mode but the entries above are
all that show up in the logs.
As of now my cluster is down.

  

sudden instability in 0.18.2

[David J. O'Dell]

We've been running 0.18.2 for over a month on an 8 node cluster.
Last week we added 4 more nodes to the cluster and have experienced 2
failures to the tasktrackers since then.
The namenodes are running fine but all jobs submitted will die when
submitted with this error on the tasktrackers.

2009-01-28 08:07:55,556 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction: attempt_200901280756_0012_m_74_2
2009-01-28 08:07:55,682 WARN org.apache.hadoop.mapred.TaskRunner:
attempt_200901280756_0012_m_74_2 Child Error
java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:462)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:403)

I tried running the tasktrackers in debug mode but the entries above are
all that show up in the logs.
As of now my cluster is down.

-- 
David O'Dell
Director, Operations
e: dod...@videoegg.com
t:  (415) 738-5152
180 Townsend St., Third Floor
San Francisco, CA 94107 

Hadoop 0.19, Cascading 1.0 and MultipleOutputs problem

[Mikhail Yakshin]

Hi,

We have a system based on Hadoop 0.18 / Cascading 0.8.1 and now I'm
trying to port it to Hadoop 0.19 / Cascading 1.0. The first serious
problem I've got into that we're extensively using MultipleOutputs in
our jobs dealing with sequence files that store Cascading's Tuples.

Since Cascading 0.9, Tuples stopped being WritableComparable and
implemented generic Hadoop serialization interface and framework.
However, in Hadoop 0.19, MultipleOutputs require use of older
WritableComparable interface. Thus, trying to do something like:

MultipleOutputs.addNamedOutput(conf, "output-name",
MySpecialMultiSplitOutputFormat.class, Tuple.class, Tuple.class);
mos = new MultipleOutputs(conf);
...
mos.getCollector("output-name", reporter).collect(tuple1, tuple2);

yields an error:

java.lang.RuntimeException: java.lang.RuntimeException: class
cascading.tuple.Tuple not org.apache.hadoop.io.WritableComparable
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:752)
at 
org.apache.hadoop.mapred.lib.MultipleOutputs.getNamedOutputKeyClass(MultipleOutputs.java:252)
at 
org.apache.hadoop.mapred.lib.MultipleOutputs$InternalFileOutputFormat.getRecordWriter(MultipleOutputs.java:556)
at 
org.apache.hadoop.mapred.lib.MultipleOutputs.getRecordWriter(MultipleOutputs.java:425)
at 
org.apache.hadoop.mapred.lib.MultipleOutputs.getCollector(MultipleOutputs.java:511)
at 
org.apache.hadoop.mapred.lib.MultipleOutputs.getCollector(MultipleOutputs.java:476)
at my.namespace.MyReducer.reduce(MyReducer.java:xxx)

Is there any known workaround for that? Any progress going on to make
MultipleOutputs use generic Hadoop serialization?

-- 
WBR, Mikhail Yakshin

Re: Using HDFS for common purpose

[Rasit OZDAS]

Thanks for responses,

Sorry, I made a mistake, it's actually not a db what I wanted. We need a
simple storage for files. Only get and put commands are enough (no queries
needed). We don't even need append, chmod, etc.

Probably from a thread on this list, I came across a link to a KFS-HDFS
comparison:
http://deliberateambiguity.typepad.com/blog/2007/10/advantages-of-k.html

It's good, that KFS is written in C++, but handling errors in C++ is usually
more difficult.
I need your opinion about which one could best fit.

Thanks,
Rasit

2009/1/27 Jim Twensky 

> You may also want to have a look at this to reach a decision based on your
> needs:
>
> http://www.swaroopch.com/notes/Distributed_Storage_Systems
>
> Jim
>
> On Tue, Jan 27, 2009 at 1:22 PM, Jim Twensky 
> wrote:
>
> > Rasit,
> >
> > What kind of data will you be storing on Hbase or directly on HDFS? Do
> you
> > aim to use it as a data source to do some key/value lookups for small
> > strings/numbers or do you want to store larger files labeled with some
> sort
> > of a key and retrieve them during a map reduce run?
> >
> > Jim
> >
> >
> > On Tue, Jan 27, 2009 at 11:51 AM, Jonathan Gray 
> wrote:
> >
> >> Perhaps what you are looking for is HBase?
> >>
> >> http://hbase.org
> >>
> >> HBase is a column-oriented, distributed store that sits on top of HDFS
> and
> >> provides random access.
> >>
> >> JG
> >>
> >> > -Original Message-
> >> > From: Rasit OZDAS [mailto:rasitoz...@gmail.com]
> >> > Sent: Tuesday, January 27, 2009 1:20 AM
> >> > To: core-user@hadoop.apache.org
> >> > Cc: arif.yil...@uzay.tubitak.gov.tr; emre.gur...@uzay.tubitak.gov.tr;
> >> > hilal.tara...@uzay.tubitak.gov.tr; serdar.ars...@uzay.tubitak.gov.tr;
> >> > hakan.kocaku...@uzay.tubitak.gov.tr; caglar.bi...@uzay.tubitak.gov.tr
> >> > Subject: Using HDFS for common purpose
> >> >
> >> > Hi,
> >> > I wanted to ask, if HDFS is a good solution just as a distributed db
> >> > (no
> >> > running jobs, only get and put commands)
> >> > A review says that "HDFS is not designed for low latency" and besides,
> >> > it's
> >> > implemented in Java.
> >> > Do these disadvantages prevent us using it?
> >> > Or could somebody suggest a better (faster) one?
> >> >
> >> > Thanks in advance..
> >> > Rasit
> >>
> >>
> >
>



-- 
M. Raşit ÖZDAŞ

Re: Netbeans/Eclipse plugin

[Rasit OZDAS]

Both DFS viewer and job submission work on eclipse v. 3.3.2.
I've given up using Ganymede, unfortunately..

2009/1/26 Aaron Kimball 

> The Eclipse plugin (which, btw, is now part of Hadoop core in src/contrib/)
> currently is inoperable. The DFS viewer works, but the job submission code
> is broken.
>
> - Aaron
>
> On Sun, Jan 25, 2009 at 9:07 PM, Amit k. Saha 
> wrote:
>
> > On Sun, Jan 25, 2009 at 9:32 PM, Edward Capriolo 
> > wrote:
> > > On Sun, Jan 25, 2009 at 10:57 AM, vinayak katkar <
> vinaykat...@gmail.com>
> > wrote:
> > >> Any one knows Netbeans or Eclipse plugin for Hadoop Map -Reduce job. I
> > want
> > >> to make plugin for netbeans
> > >>
> > >> http://vinayakkatkar.wordpress.com
> > >> --
> > >> Vinayak Katkar
> > >> Sun Campus Ambassador
> > >> Sun Microsytems,India
> > >> COEP
> > >>
> > >
> > > There is an ecplipse plugin.
> > http://www.alphaworks.ibm.com/tech/mapreducetools
> > >
> > > Seems like some work is being done on netbeans
> > > https://nbhadoop.dev.java.net/
> >
> > I started this project. But well, its caught up in the requirements
> > gathering phase.
> >
> > @ Vinayak,
> >
> > Lets take this offline and discuss. What do you think?
> >
> >
> > Thanks,
> > Amit
> >
> > >
> > > The world needs more netbeans love.
> > >
> >
> > Definitely :-)
> >
> >
> > --
> > Amit Kumar Saha
> > http://amitksaha.blogspot.com
> > http://amitsaha.in.googlepages.com/
> > *Bangalore Open Java Users Group*:http:www.bojug.in
> >
>



-- 
M. Raşit ÖZDAŞ

Re: Number of records in a MapFile

[Rasit OZDAS]

Do you mean, without scanning all the files line by line?
I know little about implementation of hadoop, but as a programmer, I can
presume that it's not possible without a complete scan.

But I can suggest a work-around:
- compute number of records manually before putting a file to HDFS.
- Append the computed number to the filename.
- modify InputReader, so that reader appends that number to the key of every
map.

Hope this helps,
Rasit

2009/1/27 Andy Liu 

> Is there a way to programatically get the number of records in a MapFile
> without doing a complete scan?
>



-- 
M. Raşit ÖZDAŞ

Number of directories problem in MapReduce operations

[Guillaume Smet]

Hi,

For a few weeks now, we experience a rather annoying problem with a
Nutch/Hadoop installation.

It's a very simple setup: the Hadoop configuration is the default from
Nutch. The version of Hadoop is the hadoop-0.17.1 jar provided by
Nutch.

During the injection operation, we now have the following errors in
one of the MapReduce task:
2009-01-21 00:00:03,344 WARN fs.AllocatorPerContext -
org.apache.hadoop.util.DiskChecker$DiskErrorException: can not create
directory: 
/tmp/hadoop-moteur/mapred/local/taskTracker/jobcache/job_local_1/reduce_3fm7iw/output
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:73)
at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createPath(LocalDirAllocator.java:253)
at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:298)
at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at 
org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputFile.java:159)
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)

2009-01-21 00:00:03,347 WARN mapred.LocalJobRunner - job_local_1
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for
taskTracker/jobcache/job_local_1/reduce_3fm7iw/output/map_0.out
at 
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
at 
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at 
org.apache.hadoop.mapred.MapOutputFile.getInputFileForWrite(MapOutputFile.java:159)
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
2009-01-21 00:00:04,211 FATAL crawl.Injector - Injector:
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1062)
at org.apache.nutch.crawl.Injector.inject(Injector.java:160)
at org.apache.nutch.crawl.Injector.run(Injector.java:190)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Injector.main(Injector.java:180)

The fact is that the directory "reduce_3fm7iw" cannot be created
because the job_local_1/ directory already contains 32k directories
and we are limited by the filesystem.

Is there any way to configure Hadoop to limit the number of
directories created (even if it's slower) or any other solution for
this problem?

I wonder if setting dfs.max.objects is a solution to my problem but
I'm not sure of the consequences it might have.

If additional information are necessary, I'll be glad to provide them.

Thanks.

-- 
Guillaume