I've already added zookeeper lib in common.loader. On Thu, Jul 16, 2015 at 11:03 AM, Lee S <[email protected]> wrote:
> sqoop log as attachment. > > > On Thu, Jul 16, 2015 at 10:43 AM, Abraham Elmahrek <[email protected]> > wrote: > >> Are there any other errors in your sqoop.log? >> >> Also, I think you need to zookeeper client jars as well for the security >> integration. >> >> On Wed, Jul 15, 2015 at 7:12 PM, Lee S <[email protected]> wrote: >> >>> Hi abe: >>> I finally reinstalled the sqoop2 on a hadoop2.3 cluster. >>> And execute show version --all, exceptions below comes out: >>> >>> Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8 >>> Compiled by root on Wed Apr 29 10:40:43 CST 2015 >>> 0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to >>> load native-hadoop library for your platform... using builtin-java classes >>> where applicable >>> Exception has occurred during processing command >>> Exception: org.apache.sqoop.common.SqoopException Message: >>> CLIENT_0004:Unable to find valid Kerberos ticket cache (kinit) >>> >>> *And derbyrepo.log as below* >>> Loaded from >>> file:/root/lcy/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/derby-10.8.2.2.jar >>> java.vendor=Oracle Corporation >>> java.runtime.version=1.7.0_45-b18 >>> user.dir=/root >>> derby.system.home=null >>> >>> derby.stream.error.file=/root/lcy/sqoop-1.99.6-bin-hadoop200/logs/derbyrepo.log >>> Database Class Loader started - derby.database.classpath='' >>> Thu Jul 16 10:13:31 CST 2015 Thread[PurgeThread,5,main] (XID = 634), >>> (SESSIONID = 1), (DATABASE = >>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db), (DRDAID = null), >>> Cleanup action starting >>> Thu Jul 16 10:13:31 CST 2015 Thread[PurgeThread,5,main] (XID = 634), >>> (SESSIONID = 1), (DATABASE = >>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db), (DRDAID = null), >>> Failed Statement is: DELETE FROM "SQOOP"."SQ_SUBMISSION" WHERE >>> "SQS_UPDATE_DATE" < ? with 1 parameters begin parameter #1: 2015-07-15 >>> 10:13:31.646 :end parameter >>> ERROR 08000: Connection closed by unknown interrupt. >>> at >>> org.apache.derby.iapi.error.StandardException.newException(Unknown Source) >>> at >>> org.apache.derby.iapi.util.InterruptStatus.setInterrupted(Unknown Source) >>> at org.apache.derby.iapi.util.InterruptStatus.throwIf(Unknown >>> Source) >>> at >>> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.checkCancellationFlag(Unknown >>> Source) >>> at >>> org.apache.derby.impl.sql.execute.TableScanResultSet.getNextRowCore(Unknown >>> Source) >>> at >>> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown >>> Source) >>> at >>> org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown >>> Source) >>> at >>> org.apache.derby.impl.sql.execute.DeleteResultSet.collectAffectedRows(Unknown >>> Source) >>> at >>> org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.collectAffectedRows(Unknown >>> Source) >>> at >>> org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.open(Unknown >>> Source) >>> at >>> org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown >>> Source) >>> at >>> org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source) >>> at >>> org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) >>> at >>> org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown >>> Source) >>> at >>> org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown >>> Source) >>> at >>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>> at >>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>> at >>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>> at >>> org.apache.sqoop.repository.common.CommonRepositoryHandler.purgeSubmissions(CommonRepositoryHandler.java:1055) >>> at >>> org.apache.sqoop.repository.JdbcRepository$26.doIt(JdbcRepository.java:594) >>> at >>> org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:92) >>> at >>> org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63) >>> >>> On Thu, Jul 16, 2015 at 9:45 AM, Lee S <[email protected]> wrote: >>> >>>> Got it. Thanks for your clear answers. >>>> >>>> On Thu, Jul 16, 2015 at 9:39 AM, Abraham Elmahrek <[email protected]> >>>> wrote: >>>> >>>>> Sqoop2 should work against Hadoop 2.3 if it's compiled against Hadoop >>>>> 2.6. Those versions are wire compatible. If you run into any issues, post >>>>> them here and we can see exactly what's going on. >>>>> >>>>> >>>>> http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility >>>>> >>>>> On Wed, Jul 15, 2015 at 6:35 PM, Lee S <[email protected]> wrote: >>>>> >>>>>> yep, I compiled it and found that no security classes were available. >>>>>> How about I jumped the compilation of security related module. >>>>>> Cause I need to use sqoop2 in a hadoop2.3 cluster. >>>>>> >>>>>> On Thu, Jul 16, 2015 at 9:29 AM, Abraham Elmahrek <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> At the moment, Hadoop 2.6 or newer is needed. The reason is that >>>>>>> there's a dependency on some of the security classes that were >>>>>>> available in >>>>>>> later versions of Hadoop. >>>>>>> >>>>>>> On Wed, Jul 15, 2015 at 6:26 PM, Lee S <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Abe: >>>>>>>> I'll try what you say. >>>>>>>> And one more question whether sqoop2 can be compiled with hadoop2.3 >>>>>>>> ? >>>>>>>> I've tried it but seemts that one class can not be found in >>>>>>>> hadoop2.3. >>>>>>>> >>>>>>>> On Thu, Jul 16, 2015 at 4:18 AM, Abraham Elmahrek <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> So this is a bit strange. There must have been jar conflicts when >>>>>>>>> you were using sqoop2 before. I see a couple of solutions immediately: >>>>>>>>> >>>>>>>>> 1. Start fresh and recreate your jobs >>>>>>>>> 2. Dump the data in the old data base and load into a new >>>>>>>>> database >>>>>>>>> >>>>>>>>> >>>>>>>>> For option two, here's a quick over view of how to do that: >>>>>>>>> >>>>>>>>> 1. Get the DDL with DBLOOK ( >>>>>>>>> >>>>>>>>> http://db.apache.org/derby/docs/10.1/tools/rtoolsdblookexamples.html). >>>>>>>>> To get DBLook working on my mac, I had to do the following: alias >>>>>>>>> dblook='java -cp >>>>>>>>> >>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derbytools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derby.jar >>>>>>>>> org.apache.derby.tools.dblook'. Then dblook -d <jdbc conn string>. >>>>>>>>> 2. Get a data dump using SYSCS_UTIL.SYSCS_EXPORT_TABLE system >>>>>>>>> procedure from IJ or programmatically ( >>>>>>>>> http://db.apache.org/derby/docs/10.7/ref/rrefexportproc.html). >>>>>>>>> 3. Load data using SYSCS_UTIL.SYSCS_IMPORT_DATA system >>>>>>>>> procedure from IJ or programmatically ( >>>>>>>>> http://db.apache.org/derby/docs/10.7/ref/rrefimportdataproc.html >>>>>>>>> ). >>>>>>>>> >>>>>>>>> Here's a quick example of what a programmatic solution might look >>>>>>>>> like (incomplete): >>>>>>>>> >>>>>>>>> public class DerbyDump { >>>>>>>>>> private static final String SCHEMA_NAME = "SQOOP"; >>>>>>>>>> >>>>>>>>>> private static List<String> fetchTableNames(String jdbc) throws >>>>>>>>>> SQLException { >>>>>>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>>>>>> List<String> tableNames = new LinkedList<String>(); >>>>>>>>>> Statement stmt = null; >>>>>>>>>> >>>>>>>>>> try { >>>>>>>>>> stmt = conn.createStatement(); >>>>>>>>>> ResultSet rs = stmt.executeQuery("select t.tablename\n" + >>>>>>>>>> " from sys.systables t, sys.sysschemas s\n" + >>>>>>>>>> " where t.schemaid = s.schemaid\n" + >>>>>>>>>> " and t.tabletype = 'T'\n" + >>>>>>>>>> " and s.schemaname = '" + SCHEMA_NAME + "'\n" + >>>>>>>>>> " order by s.schemaname, t.tablename"); >>>>>>>>>> while (rs.next()) { >>>>>>>>>> tableNames.add(rs.getString(1)); >>>>>>>>>> } >>>>>>>>>> } finally { >>>>>>>>>> if (stmt != null) { >>>>>>>>>> stmt.close(); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> if (conn != null) { >>>>>>>>>> conn.close(); >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> return tableNames; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> private static void dumpTableToFile(String jdbc, String >>>>>>>>>> tableName, String path) throws SQLException { >>>>>>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>>>>>> PreparedStatement stmt = null; >>>>>>>>>> >>>>>>>>>> try { >>>>>>>>>> stmt = conn.prepareStatement("CALL >>>>>>>>>> SYSCS_UTIL.SYSCS_EXPORT_TABLE(?, ?, ?, null, null, 'UTF-8')"); >>>>>>>>>> stmt.setString(1, SCHEMA_NAME); >>>>>>>>>> stmt.setString(2, tableName); >>>>>>>>>> stmt.setString(3, path); >>>>>>>>>> stmt.execute(); >>>>>>>>>> } finally { >>>>>>>>>> if (stmt != null) { >>>>>>>>>> stmt.close(); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> if (conn != null) { >>>>>>>>>> conn.close(); >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> private static void loadTableToFile(String jdbc, String >>>>>>>>>> tableName, String path) throws SQLException { >>>>>>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>>>>>> PreparedStatement stmt = null; >>>>>>>>>> >>>>>>>>>> try { >>>>>>>>>> stmt = conn.prepareStatement("CALL >>>>>>>>>> SYSCS_UTIL.SYSCS_IMPORT_DATA(?, ?, null, null, ?, null, null, >>>>>>>>>> 'UTF-8', 0)"); >>>>>>>>>> stmt.setString(1, SCHEMA_NAME); >>>>>>>>>> stmt.setString(2, tableName); >>>>>>>>>> stmt.setString(3, path); >>>>>>>>>> stmt.execute(); >>>>>>>>>> } finally { >>>>>>>>>> if (stmt != null) { >>>>>>>>>> stmt.close(); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> if (conn != null) { >>>>>>>>>> conn.close(); >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> >>>>>>>>> >>>>>>>>> The above snippet would need to be compiled to read from 10.8 >>>>>>>>> databases when dumping data and 10.8.2.2 when loading data. >>>>>>>>> >>>>>>>>> -Abe >>>>>>>>> >>>>>>>>> On Wed, Jul 15, 2015 at 2:54 AM, Lee S <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Richard: >>>>>>>>>> I cant run ij and dont know where derby is installed. >>>>>>>>>> And I try to reconfigure sqoop and ran it without the >>>>>>>>>> exceptions I posted earlier. >>>>>>>>>> but with the exceptions I sent on the first email. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Jul 15, 2015 at 10:16 AM, Zhou, Richard < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> There is no need to install derby in advance. >>>>>>>>>>> >>>>>>>>>>> For the sqoop.log, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.detectRepositoryVersion(DerbyRepositoryHandler.java:196)] >>>>>>>>>>> Can't fetch repository structure version. >>>>>>>>>>> >>>>>>>>>>> Caused by: java.sql.SQLException: Schema 'SQOOP' does not exist >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This error is correct. It’s fine. As it’s your first time to >>>>>>>>>>> start a Sqoop server, it will generate DB automatically (also log >>>>>>>>>>> this >>>>>>>>>>> error) if it does not exists. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> org.apache.sqoop.common.SqoopException: COMMON_0000:Unable to >>>>>>>>>>> run specified query - CREATE TABLE "SQOOP"."SQ_INPUT" ("SQI_ID" >>>>>>>>>>> BIGINT >>>>>>>>>>> GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY >>>>>>>>>>> KEY, >>>>>>>>>>> "SQI_NAME" VARCHAR(64), "SQI_FORM" BIGINT, "SQI_INDEX" SMALLINT, >>>>>>>>>>> "SQI_TYPE" >>>>>>>>>>> VARCHAR(32), "SQI_STRMASK" BOOLEAN, "SQI_STRLENGTH" SMALLINT, >>>>>>>>>>> "SQI_ENUMVALS" VARCHAR(100),CONSTRAINT "SQOOP"."FK_SQI_SQF" FOREIGN >>>>>>>>>>> KEY >>>>>>>>>>> ("SQI_FORM") REFERENCES "SQOOP"."SQ_FORM" ("SQF_ID")) >>>>>>>>>>> >>>>>>>>>>> Caused by: java.sql.SQLSyntaxErrorException: Syntax error: >>>>>>>>>>> BOOLEAN. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This seems to be the root cause. There is “Syntax error: >>>>>>>>>>> BOOLEAN” when running generate scripts. Would you run this script >>>>>>>>>>> using >>>>>>>>>>> “ij” of derby to see whether this script runs correctly in your env? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Regards >>>>>>>>>>> >>>>>>>>>>> Richard >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *From:* Lee S [mailto:[email protected]] >>>>>>>>>>> *Sent:* Tuesday, July 14, 2015 10:56 PM >>>>>>>>>>> *To:* [email protected] >>>>>>>>>>> *Subject:* sqoop2-tool verify with exception: The database was >>>>>>>>>>> created by or upgraded by version 10.8 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi all: >>>>>>>>>>> >>>>>>>>>>> I verify the configuration with the exception below: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ERROR XSLAN: Database at >>>>>>>>>>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db has an >>>>>>>>>>> incompatible >>>>>>>>>>> format with the current version of the software. The database was >>>>>>>>>>> created >>>>>>>>>>> by or upgraded by version 10.8. >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.iapi.error.StandardException.newException(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at org.apache.derby.impl.store.raw.RawStore.boot(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> at >>>>>>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>>>>>>> Source) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It seems that db in repository is not created with derby version >>>>>>>>>>> 10.8 , but I dont know how to fix it . I've checked that derby jar >>>>>>>>>>> in >>>>>>>>>>> WEB-INF/lib is 10.8.2.2. >>>>>>>>>>> >>>>>>>>>>> I'm working with sqoop-1.99.6, any idea? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
