Just to add one thing form my experience (and please feel free to correct
me if I am wrong) that even when you increase the limit, it can't go above
3000.
Regards,
Shahab
On Mon, Apr 27, 2015 at 1:21 PM, Daniel Dai wrote:
> This is the known issue with rank implementation with counter limit. We
This is the known issue with rank implementation with counter limit. We
fixed that in tez but not MR. You can either increase the limit in MR, or
try Pig on Tez.
Thanks,
Daniel
On 4/27/15, 3:07 AM, "李运田" wrote:
>I use rank to add ID now ,but I always get error "
> FATAL [AsyncDispatcher event h
There is a limit on the number of job counters you can have -- 120 is the
default. Are you sure you need more than that?
If so, you can change the limit in the mapreduce.job.counters.limit setting.
On Mon, Apr 27, 2015 at 6:07 AM, 李运田 wrote:
> I use rank to add ID now ,but I always get error "
I use rank to add ID now ,but I always get error "
FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters:
121 max=120
at
org.apache.hadoop.mapreduce.co
Have you looked at the RANK function?
https://pig.apache.org/docs/r0.11.0/basic.html#rank
On Fri, Apr 24, 2015 at 5:15 AM, 李运田 wrote:
> I have a big data about 10T ,I want to add column number to every column
> from 1 to COUNT(my data),,I use two functions in
> http://stackoverflow.com/question
I have a big data about 10T ,I want to add column number to every column from 1
to COUNT(my data),,I use two functions in
http://stackoverflow.com/questions/9288578/how-can-i-add-row-numbers-for-rows-in-pig-or-hive
and http://www.aiuxian.com/article/p-139530.html
but I always get many file
