Hey,
I made my own UDF, but I am having trouble getting it running . Any hints?
Thanks
[had...@hadoop1 ~]$ /opt/hive/bin/hive -hiveconf hive.root.logger=INFO,console
Hive history file=/tmp/hadoop/hive_job_log_hadoop_200904292045_263570529.txt
09/04/29 20:45:28 INFO exec.HiveHistory: Hive history
file=/tmp/hadoop/hive_job_log_hadoop_200904292045_263570529.txt
hive> create temporary function UDFGEO as 'com.xxx.sysops.udf.UDFGEO';
09/04/29 20:45:34 INFO parse.ParseDriver: Parsing command:
create temporary function UDFGEO as 'com.xxx.sysops.udf.UDFGEO'
09/04/29 20:45:34 INFO parse.ParseDriver: Parse Completed
09/04/29 20:45:34 INFO parse.FunctionSemanticAnalyzer: analyze done
09/04/29 20:45:34 INFO ql.Driver: Semantic Analysis Completed
09/04/29 20:45:34 INFO ql.Driver: Starting command:
create temporary function UDFGEO as 'com.xxx.sysops.udf.UDFGEO'
geoip
OK
09/04/29 20:45:34 INFO ql.Driver: OK
Time taken: 0.421 seconds
09/04/29 20:45:34 INFO CliDriver: Time taken: 0.421 seconds
09/04/29 20:46:02 INFO CliDriver: Time taken: 0.06 seconds
hive> select UDFGEO('209.191.139.200') from bigdata limit 1 ;
09/04/29 20:46:09 INFO parse.ParseDriver: Parsing command:
select UDFGEO('209.191.139.200') from bigdata limit 1
09/04/29 20:46:09 INFO parse.ParseDriver: Parse Completed
09/04/29 20:46:09 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
09/04/29 20:46:09 INFO parse.SemanticAnalyzer: Completed phase 1 of
Semantic Analysis
09/04/29 20:46:09 INFO parse.SemanticAnalyzer: Get metadata for source tables
09/04/29 20:46:09 INFO metastore.HiveMetaStore: 0: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
09/04/29 20:46:09 INFO metastore.ObjectStore: ObjectStore, initialize called
09/04/29 20:46:09 INFO metastore.ObjectStore: found resource
jpox.properties at file:/opt/hive/conf/jpox.properties
09/04/29 20:46:12 INFO metastore.ObjectStore: Initialized ObjectStore
09/04/29 20:46:12 INFO metastore.HiveMetaStore: 0: get_table :
db=default tbl=bigdata
09/04/29 20:46:13 INFO hive.log: DDL: struct bigdata { double charge,
double clus, double dst, double enumber, double etime, double hist,
double nlb, double qxb, double rnumber, double tracks, double vertex,
double zdc}
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for subqueries
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for
destination tables
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed getting
MetaData in Semantic Analysis
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for source tables
09/04/29 20:46:13 INFO metastore.HiveMetaStore: 0: get_table :
db=default tbl=bigdata
09/04/29 20:46:13 INFO metastore.HiveMetaStore: 0: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
09/04/29 20:46:13 INFO metastore.ObjectStore: ObjectStore, initialize called
09/04/29 20:46:13 INFO metastore.ObjectStore: Initialized ObjectStore
09/04/29 20:46:13 INFO hive.log: DDL: struct bigdata { double charge,
double clus, double dst, double enumber, double etime, double hist,
double nlb, double qxb, double rnumber, double tracks, double vertex,
double zdc}
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for subqueries
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for
destination tables
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed partition pruning
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed sample pruning
09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed plan generation
09/04/29 20:46:13 INFO ql.Driver: Semantic Analysis Completed
09/04/29 20:46:13 INFO ql.Driver: Starting command:
select UDFGEO('209.191.139.200') from bigdata limit 1
Total MapReduce jobs = 1
09/04/29 20:46:13 INFO ql.Driver: Total MapReduce jobs = 1
Number of reduce tasks is set to 0 since there's no reduce operator
09/04/29 20:46:13 INFO exec.ExecDriver: Number of reduce tasks is set
to 0 since there's no reduce operator
09/04/29 20:46:13 INFO exec.ExecDriver: Adding input file
/user/hive/warehouse/bigdata
09/04/29 20:46:13 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
09/04/29 20:46:13 INFO mapred.FileInputFormat: Total input paths to process : 1
Starting Job = job_200904292034_0004, Tracking URL =
http://hadoop1.jointhegrid.local:50030/jobdetails.jsp?jobid=job_200904292034_0004
09/04/29 20:46:14 INFO exec.ExecDriver: Starting Job =
job_200904292034_0004, Tracking URL =
http://hadoop1.jointhegrid.local:50030/jobdetails.jsp?jobid=job_200904292034_0004
Kill Command = /opt/hadoop/hadoop-0.19.0/bin/../bin/hadoop job
-Dmapred.job.tracker=hadoop1.jointhegrid.local:54311 -kill
job_200904292034_0004
09/04/29 20:46:14 INFO exec.ExecDriver: Kill Command =
/opt/hadoop/hadoop-0.19.0/bin/../bin/hadoop job
-Dmapred.job.tracker=hadoop1.jointhegrid.local:54311 -kill
job_200904292034_0004
map = 0%, reduce =0%
09/04/29 20:46:15 INFO exec.ExecDriver: map = 0%, reduce =0%
map = 100%, reduce =100%
09/04/29 20:46:43 INFO exec.ExecDriver: map = 100%, reduce =100%
Ended Job = job_200904292034_0004 with errors
09/04/29 20:46:43 ERROR exec.ExecDriver: Ended Job =
job_200904292034_0004 with errors
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.ExecDriver
09/04/29 20:46:43 ERROR ql.Driver: FAILED: Execution Error, return
code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
Time taken: 34.021 seconds
09/04/29 20:46:43 INFO CliDriver: Time taken: 34.021 seconds
hive>
package com.xxx.sysops.udf;
import com.maxmind.geoip.LookupService;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;
/**
*
* @author ecapriolo
*/
public class UDFGEO extends UDF{
static LookupService ls;
static {
try {
ls= new LookupService("/usr/local/share/GeoIP/GeoIPCity.dat",
LookupService.GEOIP_MEMORY_CACHE );
} catch (java.io.IOException ex) { System.err.println("geoip");}
}
Text result= new Text();
public UDFGEO(){
}
public Text evaluate(Text t){
if (t == null)
return null;
try{
//result.set(ls.getLocation(t.toString()).region);
result.set("blabla");
} catch (Exception ex){System.err.println(ex);}
return result;
}
}
