i'm sure i close all the files in the reduce step. Any other reasons cause
this problem?
2008/6/18 Konstantin Shvachko <[EMAIL PROTECTED]>:
> Did you close those files?
> If not they may be empty.
>
>
>
> ??? wrote:
>
>> Dears,
>>
>> I use hadoop-0.16.4 to do some work and found a error which i c
Did you close those files?
If not they may be empty.
??? wrote:
Dears,
I use hadoop-0.16.4 to do some work and found a error which i can't get the
reasons.
The scenario is like this: In the reduce step, instead of using
OutputCollector to write result, i use FSDataOutputStream to write result
Got a similar error when doing a mapreduce job on the master machine.
Mapping job is ok and in the end there are the right results in my
output folder, but the reduce hangs at 17% a very long time. Found this
in one of the task logs a view times:
...
2008-06-18 17:31:02,297 INFO org.apache.hadoop
Have you considered Amazon S3? I dont know how secure your
requirements are. There are lots of companies using this for just
offsite data storage and also with EC2.
C
On Jun 17, 2008, at 6:48 PM, Kenneth Miller wrote:
All,
I'm looking for a solution that would allow me to securely us
Hi all,
I'm new to Hadoop framework, i want to know when one MapReduce task is
finished, is there any easy way to save the total number of input/output
records to some file or variables?
Thanks.
You are running out of file handles on the namenode. When this
happens, the namenode cannot receive heartbeats from datanodes because
these heartbeats arrive on a tcp/ip socket connection and the namenode
does not have any free file descriptors to accept these socket
connections. Your data is stil
HDFS uses the network topology to distribute and replicate data. An
admin has to configure a script that describes the network topology to
HDFS. This is specified by using the parameter
"topology.script.file.name" in the Configuration file. This has been
tested when nodes are on different subnets i
Dears,
I use hadoop-0.16.4 to do some work and found a error which i can't get the
reasons.
The scenario is like this: In the reduce step, instead of using
OutputCollector to write result, i use FSDataOutputStream to write result to
files on HDFS(becouse i want to split the result by some rules).
JMock also works rather well, using its cglib extensions, for mocking
out fake FileSystem implementations, if you're expecting your code to
make calls directly to the filesystem for some reason.
Brian
Matt Kent wrote:
JMock is a unit testing tool for creating mock objects. I use it to mock
th
Right. But actually as far as I have seen, we don't call FileSystem.close().
Here is a corresponding stacktrace: it seems the exception happens when the
task child vm closes the sequence file (which is out of control of the
Mapper.map() function).
java.io.IOException: Filesystem closed
at
Thank you, first tried the put from the master machine, which leads to
the error. The put from the slave machine works. Guess youre right with
the configuration parameters. Appears a bit strange to me, because the
firewall settings and the hadoop-site.xml on both machines are equal.
On Tue, 2008-
hi,
i'm new in hadoop and im just testing it at the moment.
i set up a cluster with 2 nodes and it seems like they are running
normally,
the log files of the namenode and the datanodes dont show errors.
Firewall should be set right.
but when i try to upload a file to the dfs i get following m
12 matches
Mail list logo
