Trade off between hdfs efficiency and data locality.
On Tue, May 5, 2009 at 9:37 AM, Arun C Murthy wrote:
>
> On May 5, 2009, at 4:47 AM, Christian Ulrik Søttrup wrote:
>
> Hi all,
>>
>> I have a job that creates very big local files so i need to split it to as
>> many mappers as possible. Now
On May 5, 2009, at 4:47 AM, Christian Ulrik Søttrup wrote:
Hi all,
I have a job that creates very big local files so i need to split it
to as many mappers as possible. Now the DFS block size I'm
using means that this job is only split to 3 mappers. I don't want
to change the hdfs wide bloc
Cheers, that worked.
jason hadoop wrote:
Please try -D dfs.block.size=4096000
The specification must be in bytes.
On Tue, May 5, 2009 at 4:47 AM, Christian Ulrik Søttrup wrote:
Hi all,
I have a job that creates very big local files so i need to split it to as
many mappers as possible. N
Please try -D dfs.block.size=4096000
The specification must be in bytes.
On Tue, May 5, 2009 at 4:47 AM, Christian Ulrik Søttrup wrote:
> Hi all,
>
> I have a job that creates very big local files so i need to split it to as
> many mappers as possible. Now the DFS block size I'm
> using means t
Hi all,
I have a job that creates very big local files so i need to split it to
as many mappers as possible. Now the DFS block size I'm
using means that this job is only split to 3 mappers. I don't want to
change the hdfs wide block size because it works for my other jobs.
Is there a way to g
