I've mostly done it with logging, but this JIRA may interest you if
you still wish to attach a remote debugger to tasks:
https://issues.apache.org/jira/browse/MAPREDUCE-2637
On Mon, Jul 30, 2012 at 7:28 PM, Sriram Ramachandrasekaran
<sri.ram...@gmail.com> wrote:
> Harsh,
> I was waiting to try it on my cluster before I came back to report if it
> worked or not.
> I tried it and it works. The site wide configuration worked.
> The IOUtils.conf.addResource("job.xml") does the same thing as
> GoraMapReduceUtils.setIOSerialization(), so it did not help.
>
> Thanks for the help. I still would like to know, what would be a better way
> to debug distributed map reduce jobs.
> I know I can debug stand-alone jobs quite easily, but, I would like to know
> how folks do distributed map reduce jobs debugging.
>
> Thanks again!
> -Sriram
>
>
> On Sat, Jul 28, 2012 at 6:20 AM, Sriram Ramachandrasekaran
> <sri.ram...@gmail.com> wrote:
>>
>> aah! I always thought about setting io.serializations at the job level. I
>> never thought about this. will try this site wide thing. thanks again.
>>
>> On 28 Jul 2012 06:16, "Harsh J" <ha...@cloudera.com> wrote:
>>>
>>> Ah, that may be cause the core-site.xml has the property
>>> io.serializations fully defined for Gora as well? You can do that as
>>> an alternative fix, supply a core-site.xml across tasktrackers that
>>> also carry the serialization class Gora requires. I failed to think of
>>> that as a solution.
>>>
>>> On Sat, Jul 28, 2012 at 6:04 AM, Sriram Ramachandrasekaran
>>> <sri.ram...@gmail.com> wrote:
>>> > okay. But this issue didn't present itself when run in standalone mode.
>>> > :)
>>> >
>>> > On 28 Jul 2012 06:02, "Harsh J" <ha...@cloudera.com> wrote:
>>> >>
>>> >> I find it easier to run jobs via MRUnit (http://mrunit.apache.org,
>>> >> TDD) first, or via LocalJobRunner, for debug purposes.
>>> >>
>>> >> On Sat, Jul 28, 2012 at 5:53 AM, Sriram Ramachandrasekaran
>>> >> <sri.ram...@gmail.com> wrote:
>>> >> > hello harsh,
>>> >> > thanks for your investigations. while we were debugging, I saw the
>>> >> > exact
>>> >> > thing. As you pointed out, we suspected it to be a problem. So, we
>>> >> > set
>>> >> > the
>>> >> > job conf object directly on Gora's query object.
>>> >> > It goes something like this,
>>> >> > query.setConf..(job.getConfig..())
>>> >> >
>>> >> > And, then I saw that it was not getting into creating a new object
>>> >> > at
>>> >> > getOrCreate().
>>> >> >
>>> >> > OTOH, i've not tried the job.xml thing. I should give it a try n I
>>> >> > shall
>>> >> > keep the loop posted.
>>> >> >
>>> >> > I would also like to hear about standard practices for debugging
>>> >> > distributed
>>> >> > MR tasks.
>>> >> >
>>> >> > -----
>>> >> > reply from a hh device. Pl excuse typos n lack of formatting.
>>> >> >
>>> >> > On 28 Jul 2012 03:30, "Harsh J" <ha...@cloudera.com> wrote:
>>> >> >>
>>> >> >> Hi Sriram,
>>> >> >>
>>> >> >> I suspect the following in Gora to somehow be causing this issue:
>>> >> >>
>>> >> >> IOUtils source:
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> http://svn.apache.org/viewvc/gora/trunk/gora-core/src/main/java/org/apache/gora/util/IOUtils.java?view=markup
>>> >> >> QueryBase source:
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> http://svn.apache.org/viewvc/gora/trunk/gora-core/src/main/java/org/apache/gora/query/impl/QueryBase.java?view=markup
>>> >> >>
>>> >> >> Notice that IOUtils.deserialize(…) calls expect a proper
>>> >> >> Configuration
>>> >> >> object. If not passed (i.e., if null), they call the following.
>>> >> >>
>>> >> >> 68 private static Configuration
>>> >> >> getOrCreateConf(Configuration
>>> >> >> conf)
>>> >> >> {
>>> >> >> 69 if(conf == null) {
>>> >> >> 70 if(IOUtils.conf == null) {
>>> >> >> 71 IOUtils.conf = new Configuration();
>>> >> >> 72 }
>>> >> >> 73 }
>>> >> >> 74 return conf != null ? conf : IOUtils.conf;
>>> >> >> 75 }
>>> >> >>
>>> >> >> Now QueryBase, has in its readFields method, some
>>> >> >> IOUtils.deserialize(…) calls, that seem to pass a null for the
>>> >> >> configuration object. The IOUtils.deserialize(…) method hence calls
>>> >> >> this above method, and initializes a whole new Configuration
>>> >> >> object,
>>> >> >> as the passed conf object is null.
>>> >> >>
>>> >> >> If it does that, it would not be loading the "job.xml" file
>>> >> >> contents,
>>> >> >> which is the job's config file (thats something the map task's
>>> >> >> config
>>> >> >> set alone loads, and not a file thats loaded by default). So hence,
>>> >> >> custom serializers will disappear the moment it begins using this
>>> >> >> new
>>> >> >> Configuration object.
>>> >> >>
>>> >> >> This is what you'll want to investigate and fix or notify the Gora
>>> >> >> devs about (why QueryBase#readFields uses a null object, and if it
>>> >> >> can
>>> >> >> reuse some set conf object). As a cheap hack fix, maybe doing the
>>> >> >> following will make it work in an MR environment?
>>> >> >>
>>> >> >> IOUtils.conf = new Configuration();
>>> >> >> IOUtils.conf.addResource("job.xml");
>>> >> >>
>>> >> >> I haven't tried the above, but let us know how we can be of further
>>> >> >> assistance. An ideal fix would be to only use the MapTask's
>>> >> >> provided
>>> >> >> Configuration object everywhere, somehow, and never re-create one.
>>> >> >>
>>> >> >> P.s. If you want a thread ref link to share with other devs over
>>> >> >> Gora,
>>> >> >> here it is: http://search-hadoop.com/m/BXZA4dTUFC
>>> >> >>
>>> >> >> On Fri, Jul 27, 2012 at 1:24 PM, Sriram Ramachandrasekaran
>>> >> >> <sri.ram...@gmail.com> wrote:
>>> >> >> > Hello,
>>> >> >> > I have an MR job that talks to HBase. I use Gora to talk to
>>> >> >> > HBase.
>>> >> >> > Gora
>>> >> >> > also
>>> >> >> > provides couple of classes which can be extended to write Mappers
>>> >> >> > and
>>> >> >> > Reducers, if the mappers need input from an HBase store and
>>> >> >> > Reducers
>>> >> >> > need to
>>> >> >> > write it out to an HBase store. This is the reason why I use
>>> >> >> > Gora.
>>> >> >> >
>>> >> >> > Now, when I run my MR job, I get an exception as below.
>>> >> >> > (https://issues.apache.org/jira/browse/HADOOP-3093)
>>> >> >> > java.lang.RuntimeException: java.io.IOException:
>>> >> >> > java.lang.NullPointerException
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.gora.mapreduce.GoraInputFormat.setConf(GoraInputFormat.java:115)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>>> >> >> > at
>>> >> >> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:723)
>>> >> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>> >> >> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>> >> >> > at java.security.AccessController.doPrivileged(Native Method)
>>> >> >> > at javax.security.auth.Subject.doAs(Subject.java:415)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>> >> >> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>> >> >> > Caused by: java.io.IOException: java.lang.NullPointerException
>>> >> >> > at org.apache.gora.util.IOUtils.loadFromConf(IOUtils.java:483)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.gora.mapreduce.GoraInputFormat.getQuery(GoraInputFormat.java:125)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.gora.mapreduce.GoraInputFormat.setConf(GoraInputFormat.java:112)
>>> >> >> > ... 9 more
>>> >> >> > Caused by: java.lang.NullPointerException
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.hadoop.io.serializer.SerializationFactory.getDeserializer(SerializationFactory.java:77)
>>> >> >> > at org.apache.gora.util.IOUtils.deserialize(IOUtils.java:205)
>>> >> >> > at
>>> >> >> >
>>> >> >> > org.apache.gora.query.impl.QueryBase.readFields(QueryBase.java:234)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.hadoop.io.DefaultStringifier.fromString(DefaultStringifier.java:75)
>>> >> >> > at
>>> >> >> >
>>> >> >> >
>>> >> >> > org.apache.hadoop.io.DefaultStringifier.load(DefaultStringifier.java:133)
>>> >> >> > at org.apache.gora.util.IOUtils.loadFromConf(IOUtils.java:480)
>>> >> >> > ... 11 more
>>> >> >> >
>>> >> >> > I tried the following things to work through this issue.
>>> >> >> > 0. The stack trace indicates that, when setting up a new Mapper,
>>> >> >> > it
>>> >> >> > is
>>> >> >> > unable to deserialize something. (I could not get to understand
>>> >> >> > where
>>> >> >> > it
>>> >> >> > fails).
>>> >> >> > 1. I looked around the forums and realized that serialization
>>> >> >> > options
>>> >> >> > are
>>> >> >> > not getting passed, so, I tried setting up, io.serializations
>>> >> >> > config
>>> >> >> > on
>>> >> >> > the
>>> >> >> > job.
>>> >> >> > 1.1. I am not setting up the "io.serializations" myself, I use
>>> >> >> > GoraMapReduceUtils.setIOSerializations() to do it. I verified
>>> >> >> > that,
>>> >> >> > the
>>> >> >> > confs are getting proper serializers.
>>> >> >> > 2. I verified in the job xml to see if these confs have got
>>> >> >> > through,
>>> >> >> > they
>>> >> >> > were. But, it failed again.
>>> >> >> > 3. I tried starting the hadoop job runner with debug options
>>> >> >> > turned
>>> >> >> > on
>>> >> >> > and
>>> >> >> > in suspend mode, -XDebug suspend=y and I also set the VM options
>>> >> >> > for
>>> >> >> > mapred
>>> >> >> > child tasks, via the mapred.child.java.opts to see if I can debug
>>> >> >> > the
>>> >> >> > VM
>>> >> >> > that gets spawned newly. Although I get a message on my stdout
>>> >> >> > saying,
>>> >> >> > opening port X and waiting, when I try to attach a remote
>>> >> >> > debugger on
>>> >> >> > that
>>> >> >> > port, it does not work.
>>> >> >> >
>>> >> >> > I understand that, when SerializationFactory tries to deSerialize
>>> >> >> > 'something', it does not find an appropriate unmarshaller and so
>>> >> >> > it
>>> >> >> > fails.
>>> >> >> > But, I would like to know a way to find that 'something' and I
>>> >> >> > would
>>> >> >> > like to
>>> >> >> > get some idea on how (pseudo) distributed MR jobs should be
>>> >> >> > generally
>>> >> >> > debugged. I tried searching, did not find anything useful.
>>> >> >> >
>>> >> >> > Any help/pointers would be greatly useful.
>>> >> >> >
>>> >> >> > Thanks!
>>> >> >> >
>>> >> >> > --
>>> >> >> > It's just about how deep your longing is!
>>> >> >> >
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> Harsh J
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Harsh J
>>>
>>>
>>>
>>> --
>>> Harsh J
>
>
>
>
> --
> It's just about how deep your longing is!
>
--
Harsh J
