Gopal,
Here is basically my code and I can clearly see configure() was not called and
JavaCode on GenericUDF#configure reads: "This is only called in runtime of
MapRedTask.". Also based on my observation, the query is not executed as a M/R
because Yarn monitoring knows nothing about the job. Seems to be the query is
executed locally (forgive me for knowing very little on how Tez works
internally).
===========================================
public class GenericUDFTracker extends GenericUDF {
private MapredContext context;
@Override
public void configure(MapredContext aContext) {
super.configure(aContext);
context = aContext;
LOGGER.info("configure setting context:{}", context); //this never
executed
}
@Override
public Object evaluate(DeferredObject[] arguments)
throws HiveException {
LOGGER.info(ArrayUtils.toString(arguments));
String arg0 = elementOI0.getPrimitiveJavaObject(arguments[0].get());
String arg1 = elementOI1.getPrimitiveJavaObject(arguments[1].get());
LOGGER.info("context = {}", context); //it prints 'context = null'
Reporter reporter = context.getReporter(); //NPE is thrown
LOGGER.info("reporter = {}", reporter);
...
}
}
-----Original Message-----
From: Gopal Vijayaraghavan [mailto:[email protected]] On Behalf Of Gopal
Vijayaraghavan
Sent: Tuesday, April 21, 2015 11:41 PM
To: [email protected]
Subject: Re: MapredContext not available when tez enabled
> A bit digging shows that GenericUDF#configure method was not called.
>So in this case, is it possible to get counters through other means, or
>we have to implement Counter concept ourselves?
You should be getting a TezContext object there (which inherits from
MapRedContext).
And the method should get called depending on a needConfigure() check - if it
is not getting called, that is very strange.
Cheers,
Gopal
