I'm finally getting back to this. Trying to get counter information looked promising. I tried it, but my workflow job fails with exception below. I'll keep looking over my syntax, but does anyone know if maybe this feature is limited to map/reduce actions? The action I'm using is: <sqoop xmlns="uri:oozie:sqoop-action:0.2">
Any advice on what to check or try is appreciated. Thanks. ACTION[0035106-120705224209703-oozie-oozi-W@test_counters] Exception in ActionStartCommand java.lang.IllegalArgumentException: Hadoop counters not available for action [sqoop_task] at org.apache.oozie.action.hadoop.HadoopELFunctions.getCounters(HadoopELFunctions.java:57) at org.apache.oozie.action.hadoop.HadoopELFunctions.hadoop_counters(HadoopELFunctions.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.el.FunctionInvocation.evaluate(FunctionInvocation.java:172) at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:140) at org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114) at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:274) at org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:190) at org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:199) at org.apache.oozie.command.wf.ActionStartCommand.call(ActionStartCommand.java:94) at org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java:249) at org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.java:47) at org.apache.oozie.command.Command.call(Command.java:202) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:132) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Steve Charles Sogeti USA 3636 N. Central Ave. #810 Phoenix, AZ 85012 www.us.sogeti.com <http://www.us.sogeti.com/> ________________________________ From: Jarek Jarcec Cecho [mailto:[email protected]] Sent: Wed 7/4/2012 12:19 AM To: [email protected] Cc: [email protected] Subject: Re: question on oozie workflow and log output Hi Steve, I'm not sure how to process log output, but oozie do have the ability to work with counters. There is EL function hadoop:counters(action name) that will return all counters for given action. You can find examples in the documentation: http://incubator.apache.org/oozie/docs/3.2.0-incubating/docs/WorkflowFunctionalSpec.html Those counters are available even for the sqoop action if I remember correctly. Jarcec On Tue, Jul 03, 2012 at 04:29:38PM -0400, [email protected] wrote: > Does anyone know how to make use of log output from one workflow action in a > following action? > Specifically, I have a sqoop action to import data into the HDFS. Following > that action (if ok) I want to make use of information like rows/bytes > imported in the sqoop action. I can, via the oozie console, look at the log > file for the sqoop action and scrolling down, I see something like: > > Job complete: job_201205212157_421694 > Counters: 12 > Job Counters > SLOTS_MILLIS_MAPS=4794 > Total time spent by all reduces waiting after reserving slots (ms)=0 > Total time spent by all maps waiting after reserving slots (ms)=0 > Launched map tasks=1 > SLOTS_MILLIS_REDUCES=0 > FileSystemCounters > HDFS_BYTES_READ=87 > FILE_BYTES_WRITTEN=79795 > HDFS_BYTES_WRITTEN=95560 > Map-Reduce Framework > Map input records=1000 > Spilled Records=0 > Map output records=1000 > SPLIT_RAW_BYTES=87 > > So my thought is, to have a follow up java action, pass it the path to the > log file, and have the java class extract the data I want, and update a > results table. > I also considered whether I could direct the sqoop action to output > information to a properties file. Oozie does NOT like the <capture-output> > tag in the sqoop action. > Any help on this is appreciated. > Thanks. > > > Steve Charles > Senior Consultant > > Sogeti USA > 3636 N. Central Ave. > #810 > Phoenix, AZ 85012 > www.us.sogeti.com <http://www.us.sogeti.com/> >
