Does anyone know how to make use of log output from one workflow action in a
following action?
Specifically, I have a sqoop action to import data into the HDFS. Following
that action (if ok) I want to make use of information like rows/bytes imported
in the sqoop action. I can, via the oozie console, look at the log file for the
sqoop action and scrolling down, I see something like:
Job complete: job_201205212157_421694
Counters: 12
Job Counters
SLOTS_MILLIS_MAPS=4794
Total time spent by all reduces waiting after reserving slots (ms)=0
Total time spent by all maps waiting after reserving slots (ms)=0
Launched map tasks=1
SLOTS_MILLIS_REDUCES=0
FileSystemCounters
HDFS_BYTES_READ=87
FILE_BYTES_WRITTEN=79795
HDFS_BYTES_WRITTEN=95560
Map-Reduce Framework
Map input records=1000
Spilled Records=0
Map output records=1000
SPLIT_RAW_BYTES=87
So my thought is, to have a follow up java action, pass it the path to the log
file, and have the java class extract the data I want, and update a results
table.
I also considered whether I could direct the sqoop action to output information
to a properties file. Oozie does NOT like the <capture-output> tag in the sqoop
action.
Any help on this is appreciated.
Thanks.
Steve Charles
Senior Consultant
Sogeti USA
3636 N. Central Ave.
#810
Phoenix, AZ 85012
www.us.sogeti.com <http://www.us.sogeti.com/>
