tim robertson wrote:
Hi all,
I am running MR which is scanning 130M records and then trying to
group them into around 64,000 files.
The Map does the grouping of the record by determining the key, and
then I use a MultipleTextOutputFormat to write the file based on the
key:
@Override
There are a file number limitation each process can open in unix/linux. The
default number in linux is 1024, you can use
ulimit -n number
to custom this limitation and
ulimit -n
to show this limitation.
Regards,
Jeremy
--
My research interests are distributed systems, parallel computing and
Thank you Jeremy
I am on Mac (10.5.5) and it is 256 by default. I will change this and
rerun before running on the cluster.
Thanks again
Tim
On Mon, Nov 24, 2008 at 8:38 AM, Jeremy Chow [EMAIL PROTECTED] wrote:
There are a file number limitation each process can open in unix/linux. The
