UDFContext NULL JobConf
Hi, in my UDF (Java) the following code throws and NPE, when executed in -x mapred mode: Configuration jobConf = UDFContext.getUDFContext().getJobConf(); System.err.println(jobConf.toString()); I did not find any useful information as why my JobConf is always null. All I find is that this is the right way to get the JobConf in a UDF and that the behavior of what is returned when running locally (jira issue). Any ideas? I am running it on a very old Hadoop version 0.20.2 Are there some known issues? Many thanks in advanced
Re: UDFContext NULL JobConf
Sure, just did https://issues.apache.org/jira/browse/PIG-3554
Kind regards
2013/10/30 Cheolsoo Park
> Indeed. Do you mind filing a jira?
> https://issues.apache.org/jira/browse/PIG
>
>
> On Wed, Oct 30, 2013 at 11:34 AM, Henning Kropp >wrote:
>
> > Wow, I was pretty sure to have tested it, but apparently not. The reason
> > for the NPE seems to be the @MonitoredUDF
> >
> > You can reproduced it with the following code:
> >
> > import java.io.IOException;
> > import java.util.concurrent.TimeUnit;
> > import org.apache.hadoop.conf.Configuration;
> > import org.apache.pig.EvalFunc;
> > import org.apache.pig.builtin.MonitoredUDF;
> > import org.apache.pig.data.Tuple;
> > import org.apache.pig.impl.util.UDFContext;
> >
> > @MonitoredUDF(timeUnit = TimeUnit.HOURS, duration = 3, intDefault = 1)
> > public class TestJobConf extends EvalFunc {
> >
> > public TestJobConf() {
> > }
> >
> > @Override
> > public Tuple exec(Tuple tuple) throws IOException {
> > if (tuple == null || tuple.size() == 0) {
> > return null;
> > }
> > Configuration jobConf;
> > jobConf = UDFContext.getUDFContext().getJobConf();
> > System.err.println(jobConf.toString());
> > return null;
> > }
> >
> > }
> >
> > Log from the Task:
> >
> > 2013-10-30 19:24:09,516 ERROR TestJobConf:
> > java.util.concurrent.ExecutionException:
> > java.lang.NullPointerException
> > 2013-10-30 19:24:10,388 ERROR TestJobConf:
> > java.util.concurrent.ExecutionException:
> > java.lang.NullPointerException
> > 2013-10-30 19:24:10,880 ERROR TestJobConf:
> > java.util.concurrent.ExecutionException:
> > java.lang.NullPointerException
> > 2013-10-30 19:24:11,512 ERROR TestJobConf:
> > java.util.concurrent.ExecutionException:
> > java.lang.NullPointerException
> > 2013-10-30 19:24:12,264 ERROR TestJobConf:
> > java.util.concurrent.ExecutionException:
> > java.lang.NullPointerException
> >
> >
> > Without the MonitoredUDF Annotation the log output is:
> >
> > Configuration: core-default.xml, core-site.xml, mapred-default.xml,
> > mapred-site.xml, hdfs-default.xml, hdfs-site.xml, .
> >
> > Seems like a bug.
> >
> >
> > 2013/10/30 Pradeep Gollakota
> >
> > > Are you able to post your UDF (or at least a sanitized version)?
> > >
> > >
> > > On Wed, Oct 30, 2013 at 10:46 AM, Henning Kropp <
> henning.kr...@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > thanks for your reply. I read about the expected behavior on the
> > > front-end
> > > > and I am getting the NPE on the back-end. The Mappers log the
> Exception
> > > > during Execution.
> > > >
> > > > I am currently digging through debug messages. What to look out for?
> > > There
> > > > are bunch of
> > > >
> > > > [main] DEBUG org.apache.hadoop.conf.Configuration -
> > java.io.IOException:
> > > > config()
> > > >
> > > > log messages. But I recall them as being "normal" for reasons I don't
> > > > remember.
> > > >
> > > > Regards
> > > >
> > > >
> > > > 2013/10/30 Cheolsoo Park
> > > >
> > > > > Hi,
> > > > >
> > > > > Are you getting NPE on the front-end or the back-end? Sounds like
> > > jobConf
> > > > > is not added to UDFContext, which is expected on the front-end.
> > Please
> > > > see
> > > > > the comments in getJobConf() and addJobConf() in the source code:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/UDFContext.java
> > > > >
> > > > > Thanks,
> > > > > Cheolsoo
> > > > >
> > > > >
> > > > > On Wed, Oct 30, 2013 at 9:57 AM, Henning Kropp <
> > > henning.kr...@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am stuck. In my UDF (Java) extends EvalFunc the following code
> > > throws
> > > > > and
> > > > > > NPE in exec(), when executed in -x mapred mode:
> > > > > >
> > > > > > Configuration jobConf = UDFContext.getUDFContext().getJobConf();
> > > > > > System.err.println(jobConf.toString());
> > > > > >
> > > > > > I did not find any useful information as why my JobConf is always
> > > null.
> > > > > All
> > > > > > I find is that this is the right way to get the JobConf in a UDF
> > and
> > > > that
> > > > > > the behavior of what is returned when running locally (jira
> issue).
> > > > > >
> > > > > > Any ideas? I am running it on a very old Hadoop version 0.20.2
> Are
> > > > there
> > > > > > some known issues? I use Pig 0.11.1
> > > > > >
> > > > > > Many thanks in advanced
> > > > > >
> > > > > > PS: Just found someone with the same issue
> > > > > >
> > > >
> > http://stackoverflow.com/questions/18795008/accessing-hdfs-from-pig-udf
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: UDFContext NULL JobConf
Wow, I was pretty sure to have tested it, but apparently not. The reason
for the NPE seems to be the @MonitoredUDF
You can reproduced it with the following code:
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import org.apache.hadoop.conf.Configuration;
import org.apache.pig.EvalFunc;
import org.apache.pig.builtin.MonitoredUDF;
import org.apache.pig.data.Tuple;
import org.apache.pig.impl.util.UDFContext;
@MonitoredUDF(timeUnit = TimeUnit.HOURS, duration = 3, intDefault = 1)
public class TestJobConf extends EvalFunc {
public TestJobConf() {
}
@Override
public Tuple exec(Tuple tuple) throws IOException {
if (tuple == null || tuple.size() == 0) {
return null;
}
Configuration jobConf;
jobConf = UDFContext.getUDFContext().getJobConf();
System.err.println(jobConf.toString());
return null;
}
}
Log from the Task:
2013-10-30 19:24:09,516 ERROR TestJobConf:
java.util.concurrent.ExecutionException:
java.lang.NullPointerException
2013-10-30 19:24:10,388 ERROR TestJobConf:
java.util.concurrent.ExecutionException:
java.lang.NullPointerException
2013-10-30 19:24:10,880 ERROR TestJobConf:
java.util.concurrent.ExecutionException:
java.lang.NullPointerException
2013-10-30 19:24:11,512 ERROR TestJobConf:
java.util.concurrent.ExecutionException:
java.lang.NullPointerException
2013-10-30 19:24:12,264 ERROR TestJobConf:
java.util.concurrent.ExecutionException:
java.lang.NullPointerException
Without the MonitoredUDF Annotation the log output is:
Configuration: core-default.xml, core-site.xml, mapred-default.xml,
mapred-site.xml, hdfs-default.xml, hdfs-site.xml, .
Seems like a bug.
2013/10/30 Pradeep Gollakota
> Are you able to post your UDF (or at least a sanitized version)?
>
>
> On Wed, Oct 30, 2013 at 10:46 AM, Henning Kropp >wrote:
>
> > Hi,
> >
> > thanks for your reply. I read about the expected behavior on the
> front-end
> > and I am getting the NPE on the back-end. The Mappers log the Exception
> > during Execution.
> >
> > I am currently digging through debug messages. What to look out for?
> There
> > are bunch of
> >
> > [main] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException:
> > config()
> >
> > log messages. But I recall them as being "normal" for reasons I don't
> > remember.
> >
> > Regards
> >
> >
> > 2013/10/30 Cheolsoo Park
> >
> > > Hi,
> > >
> > > Are you getting NPE on the front-end or the back-end? Sounds like
> jobConf
> > > is not added to UDFContext, which is expected on the front-end. Please
> > see
> > > the comments in getJobConf() and addJobConf() in the source code:
> > >
> > >
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/UDFContext.java
> > >
> > > Thanks,
> > > Cheolsoo
> > >
> > >
> > > On Wed, Oct 30, 2013 at 9:57 AM, Henning Kropp <
> henning.kr...@gmail.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > I am stuck. In my UDF (Java) extends EvalFunc the following code
> throws
> > > and
> > > > NPE in exec(), when executed in -x mapred mode:
> > > >
> > > > Configuration jobConf = UDFContext.getUDFContext().getJobConf();
> > > > System.err.println(jobConf.toString());
> > > >
> > > > I did not find any useful information as why my JobConf is always
> null.
> > > All
> > > > I find is that this is the right way to get the JobConf in a UDF and
> > that
> > > > the behavior of what is returned when running locally (jira issue).
> > > >
> > > > Any ideas? I am running it on a very old Hadoop version 0.20.2 Are
> > there
> > > > some known issues? I use Pig 0.11.1
> > > >
> > > > Many thanks in advanced
> > > >
> > > > PS: Just found someone with the same issue
> > > >
> > http://stackoverflow.com/questions/18795008/accessing-hdfs-from-pig-udf
> > > >
> > >
> >
>
Re: UDFContext NULL JobConf
Hi, thanks for your reply. I read about the expected behavior on the front-end and I am getting the NPE on the back-end. The Mappers log the Exception during Execution. I am currently digging through debug messages. What to look out for? There are bunch of [main] DEBUG org.apache.hadoop.conf.Configuration - java.io.IOException: config() log messages. But I recall them as being "normal" for reasons I don't remember. Regards 2013/10/30 Cheolsoo Park > Hi, > > Are you getting NPE on the front-end or the back-end? Sounds like jobConf > is not added to UDFContext, which is expected on the front-end. Please see > the comments in getJobConf() and addJobConf() in the source code: > > > https://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/UDFContext.java > > Thanks, > Cheolsoo > > > On Wed, Oct 30, 2013 at 9:57 AM, Henning Kropp >wrote: > > > Hi, > > > > I am stuck. In my UDF (Java) extends EvalFunc the following code throws > and > > NPE in exec(), when executed in -x mapred mode: > > > > Configuration jobConf = UDFContext.getUDFContext().getJobConf(); > > System.err.println(jobConf.toString()); > > > > I did not find any useful information as why my JobConf is always null. > All > > I find is that this is the right way to get the JobConf in a UDF and that > > the behavior of what is returned when running locally (jira issue). > > > > Any ideas? I am running it on a very old Hadoop version 0.20.2 Are there > > some known issues? I use Pig 0.11.1 > > > > Many thanks in advanced > > > > PS: Just found someone with the same issue > > http://stackoverflow.com/questions/18795008/accessing-hdfs-from-pig-udf > > >
UDFContext NULL JobConf
Hi, I am stuck. In my UDF (Java) extends EvalFunc the following code throws and NPE in exec(), when executed in -x mapred mode: Configuration jobConf = UDFContext.getUDFContext().getJobConf(); System.err.println(jobConf.toString()); I did not find any useful information as why my JobConf is always null. All I find is that this is the right way to get the JobConf in a UDF and that the behavior of what is returned when running locally (jira issue). Any ideas? I am running it on a very old Hadoop version 0.20.2 Are there some known issues? I use Pig 0.11.1 Many thanks in advanced PS: Just found someone with the same issue http://stackoverflow.com/questions/18795008/accessing-hdfs-from-pig-udf
