My problem is same as this link http://qnalist.com/questions/6024365/1-99-5-and-unable-to-find-valid-kerberos-ticket
On Thu, Jul 16, 2015 at 11:27 AM, Lee S <[email protected]> wrote: > I've already added zookeeper lib in common.loader. > > On Thu, Jul 16, 2015 at 11:03 AM, Lee S <[email protected]> wrote: > >> sqoop log as attachment. >> >> >> On Thu, Jul 16, 2015 at 10:43 AM, Abraham Elmahrek <[email protected]> >> wrote: >> >>> Are there any other errors in your sqoop.log? >>> >>> Also, I think you need to zookeeper client jars as well for the security >>> integration. >>> >>> On Wed, Jul 15, 2015 at 7:12 PM, Lee S <[email protected]> wrote: >>> >>>> Hi abe: >>>> I finally reinstalled the sqoop2 on a hadoop2.3 cluster. >>>> And execute show version --all, exceptions below comes out: >>>> >>>> Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8 >>>> Compiled by root on Wed Apr 29 10:40:43 CST 2015 >>>> 0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to >>>> load native-hadoop library for your platform... using builtin-java classes >>>> where applicable >>>> Exception has occurred during processing command >>>> Exception: org.apache.sqoop.common.SqoopException Message: >>>> CLIENT_0004:Unable to find valid Kerberos ticket cache (kinit) >>>> >>>> *And derbyrepo.log as below* >>>> Loaded from >>>> file:/root/lcy/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/derby-10.8.2.2.jar >>>> java.vendor=Oracle Corporation >>>> java.runtime.version=1.7.0_45-b18 >>>> user.dir=/root >>>> derby.system.home=null >>>> >>>> derby.stream.error.file=/root/lcy/sqoop-1.99.6-bin-hadoop200/logs/derbyrepo.log >>>> Database Class Loader started - derby.database.classpath='' >>>> Thu Jul 16 10:13:31 CST 2015 Thread[PurgeThread,5,main] (XID = 634), >>>> (SESSIONID = 1), (DATABASE = >>>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db), (DRDAID = null), >>>> Cleanup action starting >>>> Thu Jul 16 10:13:31 CST 2015 Thread[PurgeThread,5,main] (XID = 634), >>>> (SESSIONID = 1), (DATABASE = >>>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db), (DRDAID = null), >>>> Failed Statement is: DELETE FROM "SQOOP"."SQ_SUBMISSION" WHERE >>>> "SQS_UPDATE_DATE" < ? with 1 parameters begin parameter #1: 2015-07-15 >>>> 10:13:31.646 :end parameter >>>> ERROR 08000: Connection closed by unknown interrupt. >>>> at >>>> org.apache.derby.iapi.error.StandardException.newException(Unknown Source) >>>> at >>>> org.apache.derby.iapi.util.InterruptStatus.setInterrupted(Unknown Source) >>>> at org.apache.derby.iapi.util.InterruptStatus.throwIf(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.checkCancellationFlag(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.sql.execute.TableScanResultSet.getNextRowCore(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.sql.execute.DeleteResultSet.collectAffectedRows(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.collectAffectedRows(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.open(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source) >>>> at >>>> org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source) >>>> at >>>> org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown >>>> Source) >>>> at >>>> org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown >>>> Source) >>>> at >>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>>> at >>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>>> at >>>> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105) >>>> at >>>> org.apache.sqoop.repository.common.CommonRepositoryHandler.purgeSubmissions(CommonRepositoryHandler.java:1055) >>>> at >>>> org.apache.sqoop.repository.JdbcRepository$26.doIt(JdbcRepository.java:594) >>>> at >>>> org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:92) >>>> at >>>> org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63) >>>> >>>> On Thu, Jul 16, 2015 at 9:45 AM, Lee S <[email protected]> wrote: >>>> >>>>> Got it. Thanks for your clear answers. >>>>> >>>>> On Thu, Jul 16, 2015 at 9:39 AM, Abraham Elmahrek <[email protected]> >>>>> wrote: >>>>> >>>>>> Sqoop2 should work against Hadoop 2.3 if it's compiled against Hadoop >>>>>> 2.6. Those versions are wire compatible. If you run into any issues, post >>>>>> them here and we can see exactly what's going on. >>>>>> >>>>>> >>>>>> http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility >>>>>> >>>>>> On Wed, Jul 15, 2015 at 6:35 PM, Lee S <[email protected]> wrote: >>>>>> >>>>>>> yep, I compiled it and found that no security classes were >>>>>>> available. >>>>>>> How about I jumped the compilation of security related module. >>>>>>> Cause I need to use sqoop2 in a hadoop2.3 cluster. >>>>>>> >>>>>>> On Thu, Jul 16, 2015 at 9:29 AM, Abraham Elmahrek <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> At the moment, Hadoop 2.6 or newer is needed. The reason is that >>>>>>>> there's a dependency on some of the security classes that were >>>>>>>> available in >>>>>>>> later versions of Hadoop. >>>>>>>> >>>>>>>> On Wed, Jul 15, 2015 at 6:26 PM, Lee S <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Abe: >>>>>>>>> I'll try what you say. >>>>>>>>> And one more question whether sqoop2 can be compiled with >>>>>>>>> hadoop2.3 ? >>>>>>>>> I've tried it but seemts that one class can not be found in >>>>>>>>> hadoop2.3. >>>>>>>>> >>>>>>>>> On Thu, Jul 16, 2015 at 4:18 AM, Abraham Elmahrek < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> So this is a bit strange. There must have been jar conflicts when >>>>>>>>>> you were using sqoop2 before. I see a couple of solutions >>>>>>>>>> immediately: >>>>>>>>>> >>>>>>>>>> 1. Start fresh and recreate your jobs >>>>>>>>>> 2. Dump the data in the old data base and load into a new >>>>>>>>>> database >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> For option two, here's a quick over view of how to do that: >>>>>>>>>> >>>>>>>>>> 1. Get the DDL with DBLOOK ( >>>>>>>>>> >>>>>>>>>> http://db.apache.org/derby/docs/10.1/tools/rtoolsdblookexamples.html). >>>>>>>>>> To get DBLook working on my mac, I had to do the following: alias >>>>>>>>>> dblook='java -cp >>>>>>>>>> >>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derbytools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derby.jar >>>>>>>>>> org.apache.derby.tools.dblook'. Then dblook -d <jdbc conn string>. >>>>>>>>>> 2. Get a data dump using SYSCS_UTIL.SYSCS_EXPORT_TABLE system >>>>>>>>>> procedure from IJ or programmatically ( >>>>>>>>>> http://db.apache.org/derby/docs/10.7/ref/rrefexportproc.html). >>>>>>>>>> 3. Load data using SYSCS_UTIL.SYSCS_IMPORT_DATA system >>>>>>>>>> procedure from IJ or programmatically ( >>>>>>>>>> http://db.apache.org/derby/docs/10.7/ref/rrefimportdataproc.html >>>>>>>>>> ). >>>>>>>>>> >>>>>>>>>> Here's a quick example of what a programmatic solution might look >>>>>>>>>> like (incomplete): >>>>>>>>>> >>>>>>>>>> public class DerbyDump { >>>>>>>>>>> private static final String SCHEMA_NAME = "SQOOP"; >>>>>>>>>>> >>>>>>>>>>> private static List<String> fetchTableNames(String jdbc) >>>>>>>>>>> throws SQLException { >>>>>>>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>>>>>>> List<String> tableNames = new LinkedList<String>(); >>>>>>>>>>> Statement stmt = null; >>>>>>>>>>> >>>>>>>>>>> try { >>>>>>>>>>> stmt = conn.createStatement(); >>>>>>>>>>> ResultSet rs = stmt.executeQuery("select t.tablename\n" + >>>>>>>>>>> " from sys.systables t, sys.sysschemas s\n" + >>>>>>>>>>> " where t.schemaid = s.schemaid\n" + >>>>>>>>>>> " and t.tabletype = 'T'\n" + >>>>>>>>>>> " and s.schemaname = '" + SCHEMA_NAME + "'\n" >>>>>>>>>>> + >>>>>>>>>>> " order by s.schemaname, t.tablename"); >>>>>>>>>>> while (rs.next()) { >>>>>>>>>>> tableNames.add(rs.getString(1)); >>>>>>>>>>> } >>>>>>>>>>> } finally { >>>>>>>>>>> if (stmt != null) { >>>>>>>>>>> stmt.close(); >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> if (conn != null) { >>>>>>>>>>> conn.close(); >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> return tableNames; >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> private static void dumpTableToFile(String jdbc, String >>>>>>>>>>> tableName, String path) throws SQLException { >>>>>>>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>>>>>>> PreparedStatement stmt = null; >>>>>>>>>>> >>>>>>>>>>> try { >>>>>>>>>>> stmt = conn.prepareStatement("CALL >>>>>>>>>>> SYSCS_UTIL.SYSCS_EXPORT_TABLE(?, ?, ?, null, null, 'UTF-8')"); >>>>>>>>>>> stmt.setString(1, SCHEMA_NAME); >>>>>>>>>>> stmt.setString(2, tableName); >>>>>>>>>>> stmt.setString(3, path); >>>>>>>>>>> stmt.execute(); >>>>>>>>>>> } finally { >>>>>>>>>>> if (stmt != null) { >>>>>>>>>>> stmt.close(); >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> if (conn != null) { >>>>>>>>>>> conn.close(); >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> private static void loadTableToFile(String jdbc, String >>>>>>>>>>> tableName, String path) throws SQLException { >>>>>>>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>>>>>>> PreparedStatement stmt = null; >>>>>>>>>>> >>>>>>>>>>> try { >>>>>>>>>>> stmt = conn.prepareStatement("CALL >>>>>>>>>>> SYSCS_UTIL.SYSCS_IMPORT_DATA(?, ?, null, null, ?, null, null, >>>>>>>>>>> 'UTF-8', 0)"); >>>>>>>>>>> stmt.setString(1, SCHEMA_NAME); >>>>>>>>>>> stmt.setString(2, tableName); >>>>>>>>>>> stmt.setString(3, path); >>>>>>>>>>> stmt.execute(); >>>>>>>>>>> } finally { >>>>>>>>>>> if (stmt != null) { >>>>>>>>>>> stmt.close(); >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> if (conn != null) { >>>>>>>>>>> conn.close(); >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The above snippet would need to be compiled to read from 10.8 >>>>>>>>>> databases when dumping data and 10.8.2.2 when loading data. >>>>>>>>>> >>>>>>>>>> -Abe >>>>>>>>>> >>>>>>>>>> On Wed, Jul 15, 2015 at 2:54 AM, Lee S <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Richard: >>>>>>>>>>> I cant run ij and dont know where derby is installed. >>>>>>>>>>> And I try to reconfigure sqoop and ran it without the >>>>>>>>>>> exceptions I posted earlier. >>>>>>>>>>> but with the exceptions I sent on the first email. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 15, 2015 at 10:16 AM, Zhou, Richard < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> There is no need to install derby in advance. >>>>>>>>>>>> >>>>>>>>>>>> For the sqoop.log, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.detectRepositoryVersion(DerbyRepositoryHandler.java:196)] >>>>>>>>>>>> Can't fetch repository structure version. >>>>>>>>>>>> >>>>>>>>>>>> Caused by: java.sql.SQLException: Schema 'SQOOP' does not exist >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This error is correct. It’s fine. As it’s your first time to >>>>>>>>>>>> start a Sqoop server, it will generate DB automatically (also log >>>>>>>>>>>> this >>>>>>>>>>>> error) if it does not exists. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> org.apache.sqoop.common.SqoopException: COMMON_0000:Unable to >>>>>>>>>>>> run specified query - CREATE TABLE "SQOOP"."SQ_INPUT" ("SQI_ID" >>>>>>>>>>>> BIGINT >>>>>>>>>>>> GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) >>>>>>>>>>>> PRIMARY KEY, >>>>>>>>>>>> "SQI_NAME" VARCHAR(64), "SQI_FORM" BIGINT, "SQI_INDEX" SMALLINT, >>>>>>>>>>>> "SQI_TYPE" >>>>>>>>>>>> VARCHAR(32), "SQI_STRMASK" BOOLEAN, "SQI_STRLENGTH" SMALLINT, >>>>>>>>>>>> "SQI_ENUMVALS" VARCHAR(100),CONSTRAINT "SQOOP"."FK_SQI_SQF" >>>>>>>>>>>> FOREIGN KEY >>>>>>>>>>>> ("SQI_FORM") REFERENCES "SQOOP"."SQ_FORM" ("SQF_ID")) >>>>>>>>>>>> >>>>>>>>>>>> Caused by: java.sql.SQLSyntaxErrorException: Syntax error: >>>>>>>>>>>> BOOLEAN. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This seems to be the root cause. There is “Syntax error: >>>>>>>>>>>> BOOLEAN” when running generate scripts. Would you run this script >>>>>>>>>>>> using >>>>>>>>>>>> “ij” of derby to see whether this script runs correctly in your >>>>>>>>>>>> env? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Regards >>>>>>>>>>>> >>>>>>>>>>>> Richard >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *From:* Lee S [mailto:[email protected]] >>>>>>>>>>>> *Sent:* Tuesday, July 14, 2015 10:56 PM >>>>>>>>>>>> *To:* [email protected] >>>>>>>>>>>> *Subject:* sqoop2-tool verify with exception: The database was >>>>>>>>>>>> created by or upgraded by version 10.8 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi all: >>>>>>>>>>>> >>>>>>>>>>>> I verify the configuration with the exception below: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ERROR XSLAN: Database at >>>>>>>>>>>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db has an >>>>>>>>>>>> incompatible >>>>>>>>>>>> format with the current version of the software. The database was >>>>>>>>>>>> created >>>>>>>>>>>> by or upgraded by version 10.8. >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.iapi.error.StandardException.newException(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> at >>>>>>>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>>>>>>>> Source) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> It seems that db in repository is not created with derby >>>>>>>>>>>> version 10.8 , but I dont know how to fix it . I've checked that >>>>>>>>>>>> derby jar >>>>>>>>>>>> in WEB-INF/lib is 10.8.2.2. >>>>>>>>>>>> >>>>>>>>>>>> I'm working with sqoop-1.99.6, any idea? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
