Just to add one thing form my experience (and please feel free to correct me if I am wrong) that even when you increase the limit, it can't go above 3000.
Regards, Shahab On Mon, Apr 27, 2015 at 1:21 PM, Daniel Dai <da...@hortonworks.com> wrote: > This is the known issue with rank implementation with counter limit. We > fixed that in tez but not MR. You can either increase the limit in MR, or > try Pig on Tez. > > Thanks, > Daniel > > On 4/27/15, 3:07 AM, "李运田" <cumt...@163.com> wrote: > > >I use rank to add ID now ,but I always get error " > > FATAL [AsyncDispatcher event handler] > >org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread > >org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many > >counters: 121 max=120 > > at > >org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:103) > > at > >org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:110) > > at > >org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(Abstr > >actCounterGroup.java:78) > > at > >org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(A > >bstractCounterGroup.java:95) > >" > >I get many solution in website ,but it is not resolved . perhaps there is > >something wrong with some sets like " mapreduce.job.counters.max > > mapreduce.job.counters.group.name.max > > mapreduce.job.counters.counter.name.max > > mapreduce.job.counters.groups.max" > >can you give me some advice? > > > > > > > > > > > > > > > > > >At 2015-04-24 21:56:20, "Alex Nastetsky" <alex.nastet...@vervemobile.com> > >wrote: > >>Have you looked at the RANK function? > >>https://pig.apache.org/docs/r0.11.0/basic.html#rank > >> > >>On Fri, Apr 24, 2015 at 5:15 AM, 李运田 <cumt...@163.com> wrote: > >> > >>> I have a big data about 10T ,I want to add column number to every > >>>column > >>> from 1 to COUNT(my data),,I use two functions in > >>> > >>> > http://stackoverflow.com/questions/9288578/how-can-i-add-row-numbers-for > >>>-rows-in-pig-or-hive > >>> and http://www.aiuxian.com/article/p-139530.html > >>> but I always get many files. every file column number is from 0 to > >>> ...... perhaps there are many reduces. I dont know how to add column > >>> number from 1 to COUNT(my data)? > >>> thank you ,give me some advice or some one can give me some advice > >>> websites? > >>> thank you very much. > >>> > >>> > >
