Re: Counter limit
I'm setting this property and the number of compute threads in the giraph-site.xml. This works very well. Thanks again for your help. Cheers, Christian 2013/9/10 Claudio Martella claudio.marte...@gmail.com you can set it in your giraph-site.xml, but it should work on the command line. On Tue, Sep 10, 2013 at 1:44 PM, Christian Krause m...@ckrause.org wrote: I still see the number of counters increasing in the job tracker :(. Can I also set it in my giraph-site.xml or directly in my MasterCompute class? Cheers, Christian 2013/9/10 Claudio Martella claudio.marte...@gmail.com one the command line, you can use the -D option after the GiraphRunner class before the GiraphRunner specific parameters, e.g. -D giraph. useSuperstepCounters=false On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause m...@ckrause.orgwrote: Thanks a lot. One last question: where do I set options like USE_SUPERSTEP_COUNTERS? Christian 2013/9/9 André Kelpe efeshundert...@googlemail.com On older versions of hadoop, you cannot set the counters to a higher value. That was only introduced later. I had this issue on CDH3 (~1.5 years ago) and my solution was to disable all counters for the giraph job, to make it work. If you use a more modern version of hadoop, it should be possible to increase the limit though. - André 2013/9/9 Avery Ching ach...@apache.org: If you are running out of counters, you can turn off the superstep counters /** Use superstep counters? (boolean) */ BooleanConfOption USE_SUPERSTEP_COUNTERS = new BooleanConfOption(giraph.useSuperstepCounters, true, Use superstep counters? (boolean)); On 9/9/13 6:43 AM, Claudio Martella wrote: No, I used a different counters limit on that hadoop version. Setting mapreduce.job.counters.limit to a higher number and restarting JT and TT worked for me. Maybe 64000 might be too high? Try setting it to 512. Does not look like the case, but who knows. On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause m...@ckrause.org wrote: Sorry, it still doesn't work (I ran into a different problem before I reached the limit). I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe hardcoded? Cheers Christian Am 09.09.2013 08:29 schrieb Christian Krause m...@ckrause.org: I changed the property name to mapred.job.counters.limit and restarted it again. Now it works. Thanks, Christian 2013/9/7 Claudio Martella claudio.marte...@gmail.com did you restart TT and JT? On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause m...@ckrause.org wrote: Hi, I've increased the counter limit in mapred-site.xml, but I still get the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6 Limit=50. This is my config: cat conf/mapred-site.xml ?xml version=1.0? ?xml-stylesheet type=text/xsl href=configuration.xsl? !-- Put site-specific property overrides in this file. -- configuration ... property namemapreduce.job.counters.limit/name value64000/value /property property namemapred.task.timeout/name value240/value /property ... /configuration Any ideas? Cheers, Christian -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com
Re: Counter limit
Thanks a lot. One last question: where do I set options like USE_SUPERSTEP_COUNTERS? Christian 2013/9/9 André Kelpe efeshundert...@googlemail.com On older versions of hadoop, you cannot set the counters to a higher value. That was only introduced later. I had this issue on CDH3 (~1.5 years ago) and my solution was to disable all counters for the giraph job, to make it work. If you use a more modern version of hadoop, it should be possible to increase the limit though. - André 2013/9/9 Avery Ching ach...@apache.org: If you are running out of counters, you can turn off the superstep counters /** Use superstep counters? (boolean) */ BooleanConfOption USE_SUPERSTEP_COUNTERS = new BooleanConfOption(giraph.useSuperstepCounters, true, Use superstep counters? (boolean)); On 9/9/13 6:43 AM, Claudio Martella wrote: No, I used a different counters limit on that hadoop version. Setting mapreduce.job.counters.limit to a higher number and restarting JT and TT worked for me. Maybe 64000 might be too high? Try setting it to 512. Does not look like the case, but who knows. On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause m...@ckrause.org wrote: Sorry, it still doesn't work (I ran into a different problem before I reached the limit). I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe hardcoded? Cheers Christian Am 09.09.2013 08:29 schrieb Christian Krause m...@ckrause.org: I changed the property name to mapred.job.counters.limit and restarted it again. Now it works. Thanks, Christian 2013/9/7 Claudio Martella claudio.marte...@gmail.com did you restart TT and JT? On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause m...@ckrause.org wrote: Hi, I've increased the counter limit in mapred-site.xml, but I still get the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6 Limit=50. This is my config: cat conf/mapred-site.xml ?xml version=1.0? ?xml-stylesheet type=text/xsl href=configuration.xsl? !-- Put site-specific property overrides in this file. -- configuration ... property namemapreduce.job.counters.limit/name value64000/value /property property namemapred.task.timeout/name value240/value /property ... /configuration Any ideas? Cheers, Christian -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com
Re: Counter limit
one the command line, you can use the -D option after the GiraphRunner class before the GiraphRunner specific parameters, e.g. -D giraph. useSuperstepCounters=false On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause m...@ckrause.org wrote: Thanks a lot. One last question: where do I set options like USE_SUPERSTEP_COUNTERS? Christian 2013/9/9 André Kelpe efeshundert...@googlemail.com On older versions of hadoop, you cannot set the counters to a higher value. That was only introduced later. I had this issue on CDH3 (~1.5 years ago) and my solution was to disable all counters for the giraph job, to make it work. If you use a more modern version of hadoop, it should be possible to increase the limit though. - André 2013/9/9 Avery Ching ach...@apache.org: If you are running out of counters, you can turn off the superstep counters /** Use superstep counters? (boolean) */ BooleanConfOption USE_SUPERSTEP_COUNTERS = new BooleanConfOption(giraph.useSuperstepCounters, true, Use superstep counters? (boolean)); On 9/9/13 6:43 AM, Claudio Martella wrote: No, I used a different counters limit on that hadoop version. Setting mapreduce.job.counters.limit to a higher number and restarting JT and TT worked for me. Maybe 64000 might be too high? Try setting it to 512. Does not look like the case, but who knows. On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause m...@ckrause.org wrote: Sorry, it still doesn't work (I ran into a different problem before I reached the limit). I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe hardcoded? Cheers Christian Am 09.09.2013 08:29 schrieb Christian Krause m...@ckrause.org: I changed the property name to mapred.job.counters.limit and restarted it again. Now it works. Thanks, Christian 2013/9/7 Claudio Martella claudio.marte...@gmail.com did you restart TT and JT? On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause m...@ckrause.org wrote: Hi, I've increased the counter limit in mapred-site.xml, but I still get the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6 Limit=50. This is my config: cat conf/mapred-site.xml ?xml version=1.0? ?xml-stylesheet type=text/xsl href=configuration.xsl? !-- Put site-specific property overrides in this file. -- configuration ... property namemapreduce.job.counters.limit/name value64000/value /property property namemapred.task.timeout/name value240/value /property ... /configuration Any ideas? Cheers, Christian -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com
Re: Counter limit
I still see the number of counters increasing in the job tracker :(. Can I also set it in my giraph-site.xml or directly in my MasterCompute class? Cheers, Christian 2013/9/10 Claudio Martella claudio.marte...@gmail.com one the command line, you can use the -D option after the GiraphRunner class before the GiraphRunner specific parameters, e.g. -D giraph. useSuperstepCounters=false On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause m...@ckrause.org wrote: Thanks a lot. One last question: where do I set options like USE_SUPERSTEP_COUNTERS? Christian 2013/9/9 André Kelpe efeshundert...@googlemail.com On older versions of hadoop, you cannot set the counters to a higher value. That was only introduced later. I had this issue on CDH3 (~1.5 years ago) and my solution was to disable all counters for the giraph job, to make it work. If you use a more modern version of hadoop, it should be possible to increase the limit though. - André 2013/9/9 Avery Ching ach...@apache.org: If you are running out of counters, you can turn off the superstep counters /** Use superstep counters? (boolean) */ BooleanConfOption USE_SUPERSTEP_COUNTERS = new BooleanConfOption(giraph.useSuperstepCounters, true, Use superstep counters? (boolean)); On 9/9/13 6:43 AM, Claudio Martella wrote: No, I used a different counters limit on that hadoop version. Setting mapreduce.job.counters.limit to a higher number and restarting JT and TT worked for me. Maybe 64000 might be too high? Try setting it to 512. Does not look like the case, but who knows. On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause m...@ckrause.org wrote: Sorry, it still doesn't work (I ran into a different problem before I reached the limit). I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe hardcoded? Cheers Christian Am 09.09.2013 08:29 schrieb Christian Krause m...@ckrause.org: I changed the property name to mapred.job.counters.limit and restarted it again. Now it works. Thanks, Christian 2013/9/7 Claudio Martella claudio.marte...@gmail.com did you restart TT and JT? On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause m...@ckrause.org wrote: Hi, I've increased the counter limit in mapred-site.xml, but I still get the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6 Limit=50. This is my config: cat conf/mapred-site.xml ?xml version=1.0? ?xml-stylesheet type=text/xsl href=configuration.xsl? !-- Put site-specific property overrides in this file. -- configuration ... property namemapreduce.job.counters.limit/name value64000/value /property property namemapred.task.timeout/name value240/value /property ... /configuration Any ideas? Cheers, Christian -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com
Re: Counter limit
you can set it in your giraph-site.xml, but it should work on the command line. On Tue, Sep 10, 2013 at 1:44 PM, Christian Krause m...@ckrause.org wrote: I still see the number of counters increasing in the job tracker :(. Can I also set it in my giraph-site.xml or directly in my MasterCompute class? Cheers, Christian 2013/9/10 Claudio Martella claudio.marte...@gmail.com one the command line, you can use the -D option after the GiraphRunner class before the GiraphRunner specific parameters, e.g. -D giraph. useSuperstepCounters=false On Tue, Sep 10, 2013 at 1:15 PM, Christian Krause m...@ckrause.org wrote: Thanks a lot. One last question: where do I set options like USE_SUPERSTEP_COUNTERS? Christian 2013/9/9 André Kelpe efeshundert...@googlemail.com On older versions of hadoop, you cannot set the counters to a higher value. That was only introduced later. I had this issue on CDH3 (~1.5 years ago) and my solution was to disable all counters for the giraph job, to make it work. If you use a more modern version of hadoop, it should be possible to increase the limit though. - André 2013/9/9 Avery Ching ach...@apache.org: If you are running out of counters, you can turn off the superstep counters /** Use superstep counters? (boolean) */ BooleanConfOption USE_SUPERSTEP_COUNTERS = new BooleanConfOption(giraph.useSuperstepCounters, true, Use superstep counters? (boolean)); On 9/9/13 6:43 AM, Claudio Martella wrote: No, I used a different counters limit on that hadoop version. Setting mapreduce.job.counters.limit to a higher number and restarting JT and TT worked for me. Maybe 64000 might be too high? Try setting it to 512. Does not look like the case, but who knows. On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause m...@ckrause.org wrote: Sorry, it still doesn't work (I ran into a different problem before I reached the limit). I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe hardcoded? Cheers Christian Am 09.09.2013 08:29 schrieb Christian Krause m...@ckrause.org: I changed the property name to mapred.job.counters.limit and restarted it again. Now it works. Thanks, Christian 2013/9/7 Claudio Martella claudio.marte...@gmail.com did you restart TT and JT? On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause m...@ckrause.org wrote: Hi, I've increased the counter limit in mapred-site.xml, but I still get the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6 Limit=50. This is my config: cat conf/mapred-site.xml ?xml version=1.0? ?xml-stylesheet type=text/xsl href=configuration.xsl? !-- Put site-specific property overrides in this file. -- configuration ... property namemapreduce.job.counters.limit/name value64000/value /property property namemapred.task.timeout/name value240/value /property ... /configuration Any ideas? Cheers, Christian -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com
Re: Counter limit
I changed the property name to mapred.job.counters.limit and restarted it again. Now it works. Thanks, Christian 2013/9/7 Claudio Martella claudio.marte...@gmail.com did you restart TT and JT? On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause m...@ckrause.org wrote: Hi, I've increased the counter limit in mapred-site.xml, but I still get the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6 Limit=50. This is my config: cat conf/mapred-site.xml ?xml version=1.0? ?xml-stylesheet type=text/xsl href=configuration.xsl? !-- Put site-specific property overrides in this file. -- configuration ... property namemapreduce.job.counters.limit/name value64000/value /property property namemapred.task.timeout/name value240/value /property ... /configuration Any ideas? Cheers, Christian -- Claudio Martella claudio.marte...@gmail.com
Re: Counter limit
Sorry, it still doesn't work (I ran into a different problem before I reached the limit). I am using Hadoop 0.20.203.0. Is the limit of 120 counters maybe hardcoded? Cheers Christian Am 09.09.2013 08:29 schrieb Christian Krause m...@ckrause.org: I changed the property name to mapred.job.counters.limit and restarted it again. Now it works. Thanks, Christian 2013/9/7 Claudio Martella claudio.marte...@gmail.com did you restart TT and JT? On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause m...@ckrause.org wrote: Hi, I've increased the counter limit in mapred-site.xml, but I still get the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6 Limit=50. This is my config: cat conf/mapred-site.xml ?xml version=1.0? ?xml-stylesheet type=text/xsl href=configuration.xsl? !-- Put site-specific property overrides in this file. -- configuration ... property namemapreduce.job.counters.limit/name value64000/value /property property namemapred.task.timeout/name value240/value /property ... /configuration Any ideas? Cheers, Christian -- Claudio Martella claudio.marte...@gmail.com
Re: Counter limit
If you are running out of counters, you can turn off the superstep counters /** Use superstep counters? (boolean) */ BooleanConfOption USE_SUPERSTEP_COUNTERS = new BooleanConfOption(giraph.useSuperstepCounters, true, Use superstep counters? (boolean)); On 9/9/13 6:43 AM, Claudio Martella wrote: No, I used a different counters limit on that hadoop version. Setting mapreduce.job.counters.limit to a higher number and restarting JT and TT worked for me. Maybe 64000 might be too high? Try setting it to 512. Does not look like the case, but who knows. On Mon, Sep 9, 2013 at 2:57 PM, Christian Krause m...@ckrause.org mailto:m...@ckrause.org wrote: Sorry, it still doesn't work (I ran into a different problem before I reached the limit). I am using Hadoop 0.20.203.0 tel:0.20.203.0. Is the limit of 120 counters maybe hardcoded? Cheers Christian Am 09.09.2013 08 tel:09.09.2013%2008:29 schrieb Christian Krause m...@ckrause.org mailto:m...@ckrause.org: I changed the property name to mapred.job.counters.limit and restarted it again. Now it works. Thanks, Christian 2013/9/7 Claudio Martella claudio.marte...@gmail.com mailto:claudio.marte...@gmail.com did you restart TT and JT? On Sat, Sep 7, 2013 at 7:09 AM, Christian Krause m...@ckrause.org mailto:m...@ckrause.org wrote: Hi, I've increased the counter limit in mapred-site.xml, but I still get the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6 Limit=50. This is my config: cat conf/mapred-site.xml ?xml version=1.0? ?xml-stylesheet type=text/xsl href=configuration.xsl? !-- Put site-specific property overrides in this file. -- configuration ... property namemapreduce.job.counters.limit/name value64000/value /property property namemapred.task.timeout/name value240/value /property ... /configuration Any ideas? Cheers, Christian -- Claudio Martella claudio.marte...@gmail.com mailto:claudio.marte...@gmail.com -- Claudio Martella claudio.marte...@gmail.com mailto:claudio.marte...@gmail.com
Counter limit
Hi, I've increased the counter limit in mapred-site.xml, but I still get the error: Exceeded counter limits - Counters=121 Limit=120. Groups=6 Limit=50. This is my config: cat conf/mapred-site.xml ?xml version=1.0? ?xml-stylesheet type=text/xsl href=configuration.xsl? !-- Put site-specific property overrides in this file. -- configuration ... property namemapreduce.job.counters.limit/name value64000/value /property property namemapred.task.timeout/name value240/value /property ... /configuration Any ideas? Cheers, Christian
