Are there any other errors in your sqoop.log? Also, I think you need to zookeeper client jars as well for the security integration.
On Wed, Jul 15, 2015 at 7:12 PM, Lee S <[email protected]> wrote: > Hi abe: > I finally reinstalled the sqoop2 on a hadoop2.3 cluster. > And execute show version --all, exceptions below comes out: > > Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8 > Compiled by root on Wed Apr 29 10:40:43 CST 2015 > 0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to > load native-hadoop library for your platform... using builtin-java classes > where applicable > Exception has occurred during processing command > Exception: org.apache.sqoop.common.SqoopException Message: > CLIENT_0004:Unable to find valid Kerberos ticket cache (kinit) > > *And derbyrepo.log as below* > Loaded from > file:/root/lcy/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/derby-10.8.2.2.jar > java.vendor=Oracle Corporation > java.runtime.version=1.7.0_45-b18 > user.dir=/root > derby.system.home=null > > derby.stream.error.file=/root/lcy/sqoop-1.99.6-bin-hadoop200/logs/derbyrepo.log > Database Class Loader started - derby.database.classpath='' > Thu Jul 16 10:13:31 CST 2015 Thread[PurgeThread,5,main] (XID = 634), > (SESSIONID = 1), (DATABASE = > /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db), (DRDAID = null), > Cleanup action starting > Thu Jul 16 10:13:31 CST 2015 Thread[PurgeThread,5,main] (XID = 634), > (SESSIONID = 1), (DATABASE = > /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db), (DRDAID = null), > Failed Statement is: DELETE FROM "SQOOP"."SQ_SUBMISSION" WHERE > "SQS_UPDATE_DATE" < ? with 1 parameters begin parameter #1: 2015-07-15 > 10:13:31.646 :end parameter > ERROR 08000: Connection closed by unknown interrupt. > at > org.apache.derby.iapi.error.StandardException.newException(Unknown Source) > at > org.apache.derby.iapi.util.InterruptStatus.setInterrupted(Unknown Source) > at org.apache.derby.iapi.util.InterruptStatus.throwIf(Unknown > Source) > at > org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.checkCancellationFlag(Unknown > Source) > at > org.apache.derby.impl.sql.execute.TableScanResultSet.getNextRowCore(Unknown > Source) > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) > at > org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown > Source) > at > org.apache.derby.impl.sql.execute.DeleteResultSet.collectAffectedRows(Unknown > Source) > at > org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.collectAffectedRows(Unknown > Source) > at > org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.open(Unknown > Source) > at > org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown > Source) > at > org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source) > at > org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) > at > org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown > Source) > at > org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown > Source) > at > org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) > at > org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) > at > org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) > at > org.apache.sqoop.repository.common.CommonRepositoryHandler.purgeSubmissions(CommonRepositoryHandler.java:1055) > at > org.apache.sqoop.repository.JdbcRepository$26.doIt(JdbcRepository.java:594) > at > org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:92) > at > org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63) > > On Thu, Jul 16, 2015 at 9:45 AM, Lee S <[email protected]> wrote: > >> Got it. Thanks for your clear answers. >> >> On Thu, Jul 16, 2015 at 9:39 AM, Abraham Elmahrek <[email protected]> >> wrote: >> >>> Sqoop2 should work against Hadoop 2.3 if it's compiled against Hadoop >>> 2.6. Those versions are wire compatible. If you run into any issues, post >>> them here and we can see exactly what's going on. >>> >>> >>> http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility >>> >>> On Wed, Jul 15, 2015 at 6:35 PM, Lee S <[email protected]> wrote: >>> >>>> yep, I compiled it and found that no security classes were available. >>>> How about I jumped the compilation of security related module. >>>> Cause I need to use sqoop2 in a hadoop2.3 cluster. >>>> >>>> On Thu, Jul 16, 2015 at 9:29 AM, Abraham Elmahrek <[email protected]> >>>> wrote: >>>> >>>>> At the moment, Hadoop 2.6 or newer is needed. The reason is that >>>>> there's a dependency on some of the security classes that were available >>>>> in >>>>> later versions of Hadoop. >>>>> >>>>> On Wed, Jul 15, 2015 at 6:26 PM, Lee S <[email protected]> wrote: >>>>> >>>>>> Hi Abe: >>>>>> I'll try what you say. >>>>>> And one more question whether sqoop2 can be compiled with hadoop2.3 ? >>>>>> I've tried it but seemts that one class can not be found in hadoop2.3. >>>>>> >>>>>> On Thu, Jul 16, 2015 at 4:18 AM, Abraham Elmahrek <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> So this is a bit strange. There must have been jar conflicts when >>>>>>> you were using sqoop2 before. I see a couple of solutions immediately: >>>>>>> >>>>>>> 1. Start fresh and recreate your jobs >>>>>>> 2. Dump the data in the old data base and load into a new >>>>>>> database >>>>>>> >>>>>>> >>>>>>> For option two, here's a quick over view of how to do that: >>>>>>> >>>>>>> 1. Get the DDL with DBLOOK ( >>>>>>> >>>>>>> http://db.apache.org/derby/docs/10.1/tools/rtoolsdblookexamples.html). >>>>>>> To get DBLook working on my mac, I had to do the following: alias >>>>>>> dblook='java -cp >>>>>>> >>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derbytools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derby.jar >>>>>>> org.apache.derby.tools.dblook'. Then dblook -d <jdbc conn string>. >>>>>>> 2. Get a data dump using SYSCS_UTIL.SYSCS_EXPORT_TABLE system >>>>>>> procedure from IJ or programmatically ( >>>>>>> http://db.apache.org/derby/docs/10.7/ref/rrefexportproc.html). >>>>>>> 3. Load data using SYSCS_UTIL.SYSCS_IMPORT_DATA system procedure >>>>>>> from IJ or programmatically ( >>>>>>> http://db.apache.org/derby/docs/10.7/ref/rrefimportdataproc.html >>>>>>> ). >>>>>>> >>>>>>> Here's a quick example of what a programmatic solution might look >>>>>>> like (incomplete): >>>>>>> >>>>>>> public class DerbyDump { >>>>>>>> private static final String SCHEMA_NAME = "SQOOP"; >>>>>>>> >>>>>>>> private static List<String> fetchTableNames(String jdbc) throws >>>>>>>> SQLException { >>>>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>>>> List<String> tableNames = new LinkedList<String>(); >>>>>>>> Statement stmt = null; >>>>>>>> >>>>>>>> try { >>>>>>>> stmt = conn.createStatement(); >>>>>>>> ResultSet rs = stmt.executeQuery("select t.tablename\n" + >>>>>>>> " from sys.systables t, sys.sysschemas s\n" + >>>>>>>> " where t.schemaid = s.schemaid\n" + >>>>>>>> " and t.tabletype = 'T'\n" + >>>>>>>> " and s.schemaname = '" + SCHEMA_NAME + "'\n" + >>>>>>>> " order by s.schemaname, t.tablename"); >>>>>>>> while (rs.next()) { >>>>>>>> tableNames.add(rs.getString(1)); >>>>>>>> } >>>>>>>> } finally { >>>>>>>> if (stmt != null) { >>>>>>>> stmt.close(); >>>>>>>> } >>>>>>>> >>>>>>>> if (conn != null) { >>>>>>>> conn.close(); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> return tableNames; >>>>>>>> } >>>>>>>> >>>>>>>> private static void dumpTableToFile(String jdbc, String >>>>>>>> tableName, String path) throws SQLException { >>>>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>>>> PreparedStatement stmt = null; >>>>>>>> >>>>>>>> try { >>>>>>>> stmt = conn.prepareStatement("CALL >>>>>>>> SYSCS_UTIL.SYSCS_EXPORT_TABLE(?, ?, ?, null, null, 'UTF-8')"); >>>>>>>> stmt.setString(1, SCHEMA_NAME); >>>>>>>> stmt.setString(2, tableName); >>>>>>>> stmt.setString(3, path); >>>>>>>> stmt.execute(); >>>>>>>> } finally { >>>>>>>> if (stmt != null) { >>>>>>>> stmt.close(); >>>>>>>> } >>>>>>>> >>>>>>>> if (conn != null) { >>>>>>>> conn.close(); >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> private static void loadTableToFile(String jdbc, String >>>>>>>> tableName, String path) throws SQLException { >>>>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>>>> PreparedStatement stmt = null; >>>>>>>> >>>>>>>> try { >>>>>>>> stmt = conn.prepareStatement("CALL >>>>>>>> SYSCS_UTIL.SYSCS_IMPORT_DATA(?, ?, null, null, ?, null, null, 'UTF-8', >>>>>>>> 0)"); >>>>>>>> stmt.setString(1, SCHEMA_NAME); >>>>>>>> stmt.setString(2, tableName); >>>>>>>> stmt.setString(3, path); >>>>>>>> stmt.execute(); >>>>>>>> } finally { >>>>>>>> if (stmt != null) { >>>>>>>> stmt.close(); >>>>>>>> } >>>>>>>> >>>>>>>> if (conn != null) { >>>>>>>> conn.close(); >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>> >>>>>>> The above snippet would need to be compiled to read from 10.8 >>>>>>> databases when dumping data and 10.8.2.2 when loading data. >>>>>>> >>>>>>> -Abe >>>>>>> >>>>>>> On Wed, Jul 15, 2015 at 2:54 AM, Lee S <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Richard: >>>>>>>> I cant run ij and dont know where derby is installed. >>>>>>>> And I try to reconfigure sqoop and ran it without the exceptions >>>>>>>> I posted earlier. >>>>>>>> but with the exceptions I sent on the first email. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 15, 2015 at 10:16 AM, Zhou, Richard < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> There is no need to install derby in advance. >>>>>>>>> >>>>>>>>> For the sqoop.log, >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.detectRepositoryVersion(DerbyRepositoryHandler.java:196)] >>>>>>>>> Can't fetch repository structure version. >>>>>>>>> >>>>>>>>> Caused by: java.sql.SQLException: Schema 'SQOOP' does not exist >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> This error is correct. It’s fine. As it’s your first time to start >>>>>>>>> a Sqoop server, it will generate DB automatically (also log this >>>>>>>>> error) if >>>>>>>>> it does not exists. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> org.apache.sqoop.common.SqoopException: COMMON_0000:Unable to run >>>>>>>>> specified query - CREATE TABLE "SQOOP"."SQ_INPUT" ("SQI_ID" BIGINT >>>>>>>>> GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY >>>>>>>>> KEY, >>>>>>>>> "SQI_NAME" VARCHAR(64), "SQI_FORM" BIGINT, "SQI_INDEX" SMALLINT, >>>>>>>>> "SQI_TYPE" >>>>>>>>> VARCHAR(32), "SQI_STRMASK" BOOLEAN, "SQI_STRLENGTH" SMALLINT, >>>>>>>>> "SQI_ENUMVALS" VARCHAR(100),CONSTRAINT "SQOOP"."FK_SQI_SQF" FOREIGN >>>>>>>>> KEY >>>>>>>>> ("SQI_FORM") REFERENCES "SQOOP"."SQ_FORM" ("SQF_ID")) >>>>>>>>> >>>>>>>>> Caused by: java.sql.SQLSyntaxErrorException: Syntax error: BOOLEAN. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> This seems to be the root cause. There is “Syntax error: BOOLEAN” >>>>>>>>> when running generate scripts. Would you run this script using “ij” of >>>>>>>>> derby to see whether this script runs correctly in your env? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> Richard >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *From:* Lee S [mailto:[email protected]] >>>>>>>>> *Sent:* Tuesday, July 14, 2015 10:56 PM >>>>>>>>> *To:* [email protected] >>>>>>>>> *Subject:* sqoop2-tool verify with exception: The database was >>>>>>>>> created by or upgraded by version 10.8 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi all: >>>>>>>>> >>>>>>>>> I verify the configuration with the exception below: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ERROR XSLAN: Database at >>>>>>>>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db has an incompatible >>>>>>>>> format with the current version of the software. The database was >>>>>>>>> created >>>>>>>>> by or upgraded by version 10.8. >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.iapi.error.StandardException.newException(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at org.apache.derby.impl.store.raw.RawStore.boot(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> at >>>>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>>>>> Source) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> It seems that db in repository is not created with derby version >>>>>>>>> 10.8 , but I dont know how to fix it . I've checked that derby jar in >>>>>>>>> WEB-INF/lib is 10.8.2.2. >>>>>>>>> >>>>>>>>> I'm working with sqoop-1.99.6, any idea? >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
