Hi, Do you have hadoop/conf/*in your path?
- Alex On Jun 11, 2013, at 7:09 AM, Vijay <[email protected]> wrote: > I have written a java program to extract data from HDFS file system to MySQL > database.When I run the program it tries to read it from local file system > instead of assigned HDFS file system. Can any one please analyze the below > code and let me know the issue? > > My Configurations > > Running Single Node Server in Red Hat Linux 5 > Hadoop 1.2.0 > Sqoop 1.4.3 > Running from Eclipse. > Program Follows : > > package com.archival.da; > > import java.io.IOException; > import javax.servlet.ServletException; > import javax.servlet.http.HttpServlet; > import javax.servlet.http.HttpServletRequest; > import javax.servlet.http.HttpServletResponse; > import java.sql.Connection; > import java.sql.PreparedStatement; > import java.sql.ResultSet; > import org.apache.hadoop.conf.*; > import org.apache.hadoop.fs.*; > > import com.cloudera.sqoop.*; > import com.cloudera.sqoop.tool.ExportTool; > > @SuppressWarnings("serial") > public class DataExport extends HttpServlet { > > > > @SuppressWarnings("deprecation") > public void doPost(HttpServletRequest request, HttpServletResponse response) > throws ServletException, IOException { > response.setContentType("text/html"); > > > String run_id = request.getParameter("run_id"); > Connection con=GetCon.getCon(); > PreparedStatement ps1; > try { > > String driver = "com.mysql.jdbc.Driver"; > Class.forName(driver).newInstance(); > > // Get running process Run ID to track and update status > ps1=con.prepareStatement("SELECT > POLICY.SRC_TABLE,POLICY.SRC_DB,CON.SERVER,CON.PORT,RT.RUN_DATE,CON.USER,CON.PWD > FROM POLICY JOIN CONNECTION AS CON ON POLICY.C_ID=CON.C_ID JOIN RUN_TRACKER > AS RT ON POLICY.ID=RT.POLICY_ID AND RUN_ID=?"); > ps1.setString(1,run_id); > ResultSet rs1=ps1.executeQuery(); > rs1.next(); > String tgtTable=rs1.getString(1); > String runDate=rs1.getDate(5).toString(); > String newRunDate=runDate.replace("-", "_"); > String restore_dir=tgtTable+"_"+newRunDate; > String ServerNm = > "jdbc:mysql://"+rs1.getString(3)+":"+rs1.getString(4)+"/"+rs1.getString(2); > String ConUser=rs1.getString(6); > String ConPass=rs1.getString(7); > > > Configuration config = new Configuration(); > config.addResource(new > Path("/ms/hadoop-1.2.0/conf/core-site.xml")); > config.addResource(new > Path("/ms/hadoop-1.2.0/conf/hdfs-site.xml")); > FileSystem dfs = FileSystem.get(config); > String exportDir=dfs.getWorkingDirectory()+"/"+restore_dir; > System.out.println(exportDir); > Path path = new Path(exportDir); > SqoopOptions options=new SqoopOptions(); > options.setDriverClassName(driver); > options.setHadoopMapRedHome("/ms/hadoop-1.2.0"); > options.setConnectString(ServerNm); > options.setUsername(ConUser); > options.setPassword(ConPass); > options.setExportDir(exportDir); > options.setTableName(tgtTable); > options.setInputFieldsTerminatedBy(','); > options.setNumMappers(1); > > int status=new ExportTool().run(options); > System.out.println(status); > if(status==0){ > > dfs.delete(path,true); > > > } > > > > response.getWriter().write("Restore Process Completed"); > > con.close(); > } catch (Exception e){ > e.printStackTrace(); > } > > } > > } > Console Error Message : I also printed to ensure the exportDir is assigned > with hdfs path > > hdfs://localhost:9000/user/root/city_2013_06_10 > 13/06/09 21:00:22 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in > the environment. Cannot check for additional configuration. > 13/06/09 21:00:23 WARN sqoop.ConnFactory: Parameter --driver is set to an > explicit driver however appropriate connection manager is not being set (via > --connection-manager). Sqoop is going to fall back to > org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which > connection manager should be used next time. > 13/06/09 21:00:23 INFO manager.SqlManager: Using default fetchSize of 1000 > 13/06/09 21:00:23 INFO tool.CodeGenTool: Beginning code generation > 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT > t.* FROM city AS t WHERE 1=0 > 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT > t.* FROM city AS t WHERE 1=0 > 13/06/09 21:00:23 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is > /ms/hadoop-1.2.0 > Note: /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.java uses > or overrides a deprecated API. > Note: Recompile with -Xlint:deprecation for details. > 13/06/09 21:00:23 INFO orm.CompilationManager: Writing jar file: > /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.jar > 13/06/09 21:00:23 INFO mapreduce.ExportJobBase: Beginning export of city > 13/06/09 21:00:24 INFO manager.SqlManager: Executing SQL statement: SELECT > t.* FROM city AS t WHERE 1=0 > 13/06/09 21:00:24 WARN mapreduce.JobBase: SQOOP_HOME is unset. May not be > able to find all job dependencies. > 13/06/09 21:00:25 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 13/06/09 21:00:25 WARN mapred.JobClient: Use GenericOptionsParser for parsing > the arguments. Applications should implement Tool for the same. > 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process : 1 > 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process : 1 > 13/06/09 21:00:27 WARN snappy.LoadSnappy: Snappy native library not loaded > 13/06/09 21:00:27 INFO mapred.JobClient: Cleaning up the staging area > file:/tmp/hadoop-root/mapred/staging/root1015206311/.staging/job_local1015206311_0001 > 13/06/09 21:00:27 ERROR security.UserGroupInformation: > PriviledgedActionException as:root cause:java.io.FileNotFoundException: File > /user/root/city_2013_06_10/part-m-00000 does not exist. > 13/06/09 21:00:27 ERROR tool.ExportTool: Encountered IOException running > export job: java.io.FileNotFoundException: File > /user/root/city_2013_06_10/part-m-00000 does not exist. > just addtional note..In Eclipse, it shows com.cloudera.sqoop.tool.ExportTool; > is deprecated. > > > On Tue, Jun 11, 2013 at 10:21 AM, Vijay <[email protected]> wrote: > I have written a java program to extract data from HDFS file system to MySQL > database.When I run the program it tries to read it from local file system > instead of assigned HDFS file system. Can any one please analyze the below > code and let me know the issue? > > My Configurations > > Running Single Node Server in Red Hat Linux 5 > Hadoop 1.2.0 > Sqoop 1.4.3 > Running from Eclipse. > Program Follows : > > package com.archival.da; > > import java.io.IOException; > import javax.servlet.ServletException; > import javax.servlet.http.HttpServlet; > import javax.servlet.http.HttpServletRequest; > import javax.servlet.http.HttpServletResponse; > import java.sql.Connection; > import java.sql.PreparedStatement; > import java.sql.ResultSet; > import org.apache.hadoop.conf.*; > import org.apache.hadoop.fs.*; > > import com.cloudera.sqoop.*; > import com.cloudera.sqoop.tool.ExportTool; > > @SuppressWarnings("serial") > public class DataExport extends HttpServlet { > > > > @SuppressWarnings("deprecation") > public void doPost(HttpServletRequest request, HttpServletResponse response) > throws ServletException, IOException { > response.setContentType("text/html"); > > > String run_id = request.getParameter("run_id"); > Connection con=GetCon.getCon(); > PreparedStatement ps1; > try { > > String driver = "com.mysql.jdbc.Driver"; > Class.forName(driver).newInstance(); > > // Get running process Run ID to track and update status > ps1=con.prepareStatement("SELECT > POLICY.SRC_TABLE,POLICY.SRC_DB,CON.SERVER,CON.PORT,RT.RUN_DATE,CON.USER,CON.PWD > FROM POLICY JOIN CONNECTION AS CON ON POLICY.C_ID=CON.C_ID JOIN RUN_TRACKER > AS RT ON POLICY.ID=RT.POLICY_ID AND RUN_ID=?"); > ps1.setString(1,run_id); > ResultSet rs1=ps1.executeQuery(); > rs1.next(); > String tgtTable=rs1.getString(1); > String runDate=rs1.getDate(5).toString(); > String newRunDate=runDate.replace("-", "_"); > String restore_dir=tgtTable+"_"+newRunDate; > String ServerNm = > "jdbc:mysql://"+rs1.getString(3)+":"+rs1.getString(4)+"/"+rs1.getString(2); > String ConUser=rs1.getString(6); > String ConPass=rs1.getString(7); > > > Configuration config = new Configuration(); > config.addResource(new > Path("/ms/hadoop-1.2.0/conf/core-site.xml")); > config.addResource(new > Path("/ms/hadoop-1.2.0/conf/hdfs-site.xml")); > FileSystem dfs = FileSystem.get(config); > String exportDir=dfs.getWorkingDirectory()+"/"+restore_dir; > System.out.println(exportDir); > Path path = new Path(exportDir); > SqoopOptions options=new SqoopOptions(); > options.setDriverClassName(driver); > options.setHadoopMapRedHome("/ms/hadoop-1.2.0"); > options.setConnectString(ServerNm); > options.setUsername(ConUser); > options.setPassword(ConPass); > options.setExportDir(exportDir); > options.setTableName(tgtTable); > options.setInputFieldsTerminatedBy(','); > options.setNumMappers(1); > > int status=new ExportTool().run(options); > System.out.println(status); > if(status==0){ > > dfs.delete(path,true); > > > } > > > > response.getWriter().write("Restore Process Completed"); > > con.close(); > } catch (Exception e){ > e.printStackTrace(); > } > > } > > } > Console Error Message : I also printed to ensure the exportDir is assigned > with hdfs path > > hdfs://localhost:9000/user/root/city_2013_06_10 > 13/06/09 21:00:22 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in > the environment. Cannot check for additional configuration. > 13/06/09 21:00:23 WARN sqoop.ConnFactory: Parameter --driver is set to an > explicit driver however appropriate connection manager is not being set (via > --connection-manager). Sqoop is going to fall back to > org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which > connection manager should be used next time. > 13/06/09 21:00:23 INFO manager.SqlManager: Using default fetchSize of 1000 > 13/06/09 21:00:23 INFO tool.CodeGenTool: Beginning code generation > 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT > t.* FROM city AS t WHERE 1=0 > 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT > t.* FROM city AS t WHERE 1=0 > 13/06/09 21:00:23 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is > /ms/hadoop-1.2.0 > Note: /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.java uses > or overrides a deprecated API. > Note: Recompile with -Xlint:deprecation for details. > 13/06/09 21:00:23 INFO orm.CompilationManager: Writing jar file: > /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.jar > 13/06/09 21:00:23 INFO mapreduce.ExportJobBase: Beginning export of city > 13/06/09 21:00:24 INFO manager.SqlManager: Executing SQL statement: SELECT > t.* FROM city AS t WHERE 1=0 > 13/06/09 21:00:24 WARN mapreduce.JobBase: SQOOP_HOME is unset. May not be > able to find all job dependencies. > 13/06/09 21:00:25 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 13/06/09 21:00:25 WARN mapred.JobClient: Use GenericOptionsParser for parsing > the arguments. Applications should implement Tool for the same. > 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process : 1 > 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process : 1 > 13/06/09 21:00:27 WARN snappy.LoadSnappy: Snappy native library not loaded > 13/06/09 21:00:27 INFO mapred.JobClient: Cleaning up the staging area > file:/tmp/hadoop-root/mapred/staging/root1015206311/.staging/job_local1015206311_0001 > 13/06/09 21:00:27 ERROR security.UserGroupInformation: > PriviledgedActionException as:root cause:java.io.FileNotFoundException: File > /user/root/city_2013_06_10/part-m-00000 does not exist. > 13/06/09 21:00:27 ERROR tool.ExportTool: Encountered IOException running > export job: java.io.FileNotFoundException: File > /user/root/city_2013_06_10/part-m-00000 does not exist. > -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
