Re: Send a map to all nodes

Hi Luiz, you should consider Storm or S4for your purpose. In Storm you can create topology to run your algorithm on all nodes. HTH Samir On 29 March 2012 14:58, Luiz Carlos Muniz wrote: > Do not worry about this. > > My pro

Re: Debug MR tasks impossible.

You can run a job in local mode without a cluster and run in a debugger enabling breakpoints single stepping - examination of variables ... If you do this 1) Choose a SMALL problem 2) Beware of the fact that there is only one VM - that means be cautions with static variables - they cause leakage f

RE: Debug MR tasks impossible.

You can also decompose your problem so that you have code that is testable outside the map-reduce process. This way, you can ensure all your "helper" classes/methods are behaving as you expect, and you can debug this locally. Once that is done, you are left with only debugging things related t

RE: Debug MR tasks impossible.

Hi Pedro, You can use the below property to pass the debug params to the child jvm. And also you should make sure that have only one task running at a time by giving the input appropriately. mapred.child.java.opts -Xmx200m Java opts for the task tracker child processes. The followi

Re: Debug MR tasks impossible.

On 0.23 I usually breakpoint on DefaultContainerExecutor.java : 139. Modify the script mentioned in nmPrivateContainerScriptPath to include JAVA remote debugging options. Connect to the nodemanager and then breakpoint again in MRAppMaster.java. Tedious. I know. :( On Thu, Mar 29, 2012 at 11:06 AM,

Re: Debug MR tasks impossible.

Can I suggest unit testing with MRUnit: https://cwiki.apache.org/confluence/display/MRUNIT/Index https://cwiki.apache.org/confluence/display/MRUNIT/MRUnit+Tutorial Brock On Thu, Mar 29, 2012 at 10:33 AM, Pedro Costa wrote: > Hi, > > I'm trying to debug map and reduce tasks for a quite long time

Re: Debug MR tasks impossible.

Hi Pedro, I know its sub-optimal but you should be able to put in as many System.out.println / log messages as you want and you should be able to see them in stdout, and syslog files. Which version of hadoop are you using? On Thu, Mar 29, 2012 at 10:33 AM, Pedro Costa wrote: > Hi, > > I'm try

Debug MR tasks impossible.

Hi, I'm trying to debug map and reduce tasks for a quite long time, and it seems that it's impossible. MR are launched in new process and there's no way to debug them. Even with IsolationRunner class it's impossible. This isn't good because I really need to debug the class, to understand some chan

Re: Possible to poll JobTracker for information from any language?

Ryan, Please don't cross post. On the 1.0 line I do not know of any way to get this information out for any language but java. And even then it is simplest to use the hadoop client libraries because the protocol has changed a lot from say 1.0 to 2.0 it is completely different. However if you

Re: Conf property: mapred.max.map.failures.percent

It has been deprecated and changed to mapreduce.map.failures.maxpercent. If you set the old one you will get a warning message saying to upgrade to the new one, but the old one should still work. --Bobby On 3/29/12 6:52 AM, "Ajit Ratnaparkhi" wrote: Hi, I dont see mapred.max.map.failures.pe

Re: Send a map to all nodes

Do not worry about this. My problem is just run an algorithm on all nodes in a grid. So I realized, hadoop does not serve for this purpose and I am already studying a alternative. If you have some suggestion I will be grateful. Luiz Carlos Melo Muniz 2012/3/29 Harsh J > Luiz, > > Though it