Hi abe:
I finally reinstalled the sqoop2 on a hadoop2.3 cluster.
And execute show version --all, exceptions below comes out:
Sqoop 1.99.6 source revision 07244c3915975f26f03d9e1edf09ab7d06619bb8
Compiled by root on Wed Apr 29 10:40:43 CST 2015
0 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Exception has occurred during processing command
Exception: org.apache.sqoop.common.SqoopException Message:
CLIENT_0004:Unable to find valid Kerberos ticket cache (kinit)
*And derbyrepo.log as below*
Loaded from
file:/root/lcy/sqoop-1.99.6-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/derby-10.8.2.2.jar
java.vendor=Oracle Corporation
java.runtime.version=1.7.0_45-b18
user.dir=/root
derby.system.home=null
derby.stream.error.file=/root/lcy/sqoop-1.99.6-bin-hadoop200/logs/derbyrepo.log
Database Class Loader started - derby.database.classpath=''
Thu Jul 16 10:13:31 CST 2015 Thread[PurgeThread,5,main] (XID = 634),
(SESSIONID = 1), (DATABASE =
/root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db), (DRDAID = null),
Cleanup action starting
Thu Jul 16 10:13:31 CST 2015 Thread[PurgeThread,5,main] (XID = 634),
(SESSIONID = 1), (DATABASE =
/root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db), (DRDAID = null),
Failed Statement is: DELETE FROM "SQOOP"."SQ_SUBMISSION" WHERE
"SQS_UPDATE_DATE" < ? with 1 parameters begin parameter #1: 2015-07-15
10:13:31.646 :end parameter
ERROR 08000: Connection closed by unknown interrupt.
at
org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at
org.apache.derby.iapi.util.InterruptStatus.setInterrupted(Unknown Source)
at org.apache.derby.iapi.util.InterruptStatus.throwIf(Unknown
Source)
at
org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.checkCancellationFlag(Unknown
Source)
at
org.apache.derby.impl.sql.execute.TableScanResultSet.getNextRowCore(Unknown
Source)
at
org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown
Source)
at
org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown
Source)
at
org.apache.derby.impl.sql.execute.DeleteResultSet.collectAffectedRows(Unknown
Source)
at
org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.collectAffectedRows(Unknown
Source)
at
org.apache.derby.impl.sql.execute.DeleteCascadeResultSet.open(Unknown
Source)
at
org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown
Source)
at
org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
at
org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeUpdate(Unknown
Source)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
at
org.apache.sqoop.repository.common.CommonRepositoryHandler.purgeSubmissions(CommonRepositoryHandler.java:1055)
at
org.apache.sqoop.repository.JdbcRepository$26.doIt(JdbcRepository.java:594)
at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:92)
at
org.apache.sqoop.repository.JdbcRepository.doWithConnection(JdbcRepository.java:63)
On Thu, Jul 16, 2015 at 9:45 AM, Lee S <[email protected]> wrote:
> Got it. Thanks for your clear answers.
>
> On Thu, Jul 16, 2015 at 9:39 AM, Abraham Elmahrek <[email protected]>
> wrote:
>
>> Sqoop2 should work against Hadoop 2.3 if it's compiled against Hadoop
>> 2.6. Those versions are wire compatible. If you run into any issues, post
>> them here and we can see exactly what's going on.
>>
>>
>> http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility
>>
>> On Wed, Jul 15, 2015 at 6:35 PM, Lee S <[email protected]> wrote:
>>
>>> yep, I compiled it and found that no security classes were available.
>>> How about I jumped the compilation of security related module.
>>> Cause I need to use sqoop2 in a hadoop2.3 cluster.
>>>
>>> On Thu, Jul 16, 2015 at 9:29 AM, Abraham Elmahrek <[email protected]>
>>> wrote:
>>>
>>>> At the moment, Hadoop 2.6 or newer is needed. The reason is that
>>>> there's a dependency on some of the security classes that were available in
>>>> later versions of Hadoop.
>>>>
>>>> On Wed, Jul 15, 2015 at 6:26 PM, Lee S <[email protected]> wrote:
>>>>
>>>>> Hi Abe:
>>>>> I'll try what you say.
>>>>> And one more question whether sqoop2 can be compiled with hadoop2.3 ?
>>>>> I've tried it but seemts that one class can not be found in hadoop2.3.
>>>>>
>>>>> On Thu, Jul 16, 2015 at 4:18 AM, Abraham Elmahrek <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> So this is a bit strange. There must have been jar conflicts when you
>>>>>> were using sqoop2 before. I see a couple of solutions immediately:
>>>>>>
>>>>>> 1. Start fresh and recreate your jobs
>>>>>> 2. Dump the data in the old data base and load into a new database
>>>>>>
>>>>>>
>>>>>> For option two, here's a quick over view of how to do that:
>>>>>>
>>>>>> 1. Get the DDL with DBLOOK (
>>>>>> http://db.apache.org/derby/docs/10.1/tools/rtoolsdblookexamples.html).
>>>>>> To get DBLook working on my mac, I had to do the following: alias
>>>>>> dblook='java -cp
>>>>>>
>>>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derbytools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derby.jar
>>>>>> org.apache.derby.tools.dblook'. Then dblook -d <jdbc conn string>.
>>>>>> 2. Get a data dump using SYSCS_UTIL.SYSCS_EXPORT_TABLE system
>>>>>> procedure from IJ or programmatically (
>>>>>> http://db.apache.org/derby/docs/10.7/ref/rrefexportproc.html).
>>>>>> 3. Load data using SYSCS_UTIL.SYSCS_IMPORT_DATA system procedure
>>>>>> from IJ or programmatically (
>>>>>> http://db.apache.org/derby/docs/10.7/ref/rrefimportdataproc.html).
>>>>>>
>>>>>> Here's a quick example of what a programmatic solution might look
>>>>>> like (incomplete):
>>>>>>
>>>>>> public class DerbyDump {
>>>>>>> private static final String SCHEMA_NAME = "SQOOP";
>>>>>>>
>>>>>>> private static List<String> fetchTableNames(String jdbc) throws
>>>>>>> SQLException {
>>>>>>> Connection conn = DriverManager.getConnection(jdbc);
>>>>>>> List<String> tableNames = new LinkedList<String>();
>>>>>>> Statement stmt = null;
>>>>>>>
>>>>>>> try {
>>>>>>> stmt = conn.createStatement();
>>>>>>> ResultSet rs = stmt.executeQuery("select t.tablename\n" +
>>>>>>> " from sys.systables t, sys.sysschemas s\n" +
>>>>>>> " where t.schemaid = s.schemaid\n" +
>>>>>>> " and t.tabletype = 'T'\n" +
>>>>>>> " and s.schemaname = '" + SCHEMA_NAME + "'\n" +
>>>>>>> " order by s.schemaname, t.tablename");
>>>>>>> while (rs.next()) {
>>>>>>> tableNames.add(rs.getString(1));
>>>>>>> }
>>>>>>> } finally {
>>>>>>> if (stmt != null) {
>>>>>>> stmt.close();
>>>>>>> }
>>>>>>>
>>>>>>> if (conn != null) {
>>>>>>> conn.close();
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> return tableNames;
>>>>>>> }
>>>>>>>
>>>>>>> private static void dumpTableToFile(String jdbc, String tableName,
>>>>>>> String path) throws SQLException {
>>>>>>> Connection conn = DriverManager.getConnection(jdbc);
>>>>>>> PreparedStatement stmt = null;
>>>>>>>
>>>>>>> try {
>>>>>>> stmt = conn.prepareStatement("CALL
>>>>>>> SYSCS_UTIL.SYSCS_EXPORT_TABLE(?, ?, ?, null, null, 'UTF-8')");
>>>>>>> stmt.setString(1, SCHEMA_NAME);
>>>>>>> stmt.setString(2, tableName);
>>>>>>> stmt.setString(3, path);
>>>>>>> stmt.execute();
>>>>>>> } finally {
>>>>>>> if (stmt != null) {
>>>>>>> stmt.close();
>>>>>>> }
>>>>>>>
>>>>>>> if (conn != null) {
>>>>>>> conn.close();
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> private static void loadTableToFile(String jdbc, String tableName,
>>>>>>> String path) throws SQLException {
>>>>>>> Connection conn = DriverManager.getConnection(jdbc);
>>>>>>> PreparedStatement stmt = null;
>>>>>>>
>>>>>>> try {
>>>>>>> stmt = conn.prepareStatement("CALL
>>>>>>> SYSCS_UTIL.SYSCS_IMPORT_DATA(?, ?, null, null, ?, null, null, 'UTF-8',
>>>>>>> 0)");
>>>>>>> stmt.setString(1, SCHEMA_NAME);
>>>>>>> stmt.setString(2, tableName);
>>>>>>> stmt.setString(3, path);
>>>>>>> stmt.execute();
>>>>>>> } finally {
>>>>>>> if (stmt != null) {
>>>>>>> stmt.close();
>>>>>>> }
>>>>>>>
>>>>>>> if (conn != null) {
>>>>>>> conn.close();
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>
>>>>>> The above snippet would need to be compiled to read from 10.8
>>>>>> databases when dumping data and 10.8.2.2 when loading data.
>>>>>>
>>>>>> -Abe
>>>>>>
>>>>>> On Wed, Jul 15, 2015 at 2:54 AM, Lee S <[email protected]> wrote:
>>>>>>
>>>>>>> Hi Richard:
>>>>>>> I cant run ij and dont know where derby is installed.
>>>>>>> And I try to reconfigure sqoop and ran it without the exceptions
>>>>>>> I posted earlier.
>>>>>>> but with the exceptions I sent on the first email.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 15, 2015 at 10:16 AM, Zhou, Richard <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> There is no need to install derby in advance.
>>>>>>>>
>>>>>>>> For the sqoop.log,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.detectRepositoryVersion(DerbyRepositoryHandler.java:196)]
>>>>>>>> Can't fetch repository structure version.
>>>>>>>>
>>>>>>>> Caused by: java.sql.SQLException: Schema 'SQOOP' does not exist
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This error is correct. It’s fine. As it’s your first time to start
>>>>>>>> a Sqoop server, it will generate DB automatically (also log this
>>>>>>>> error) if
>>>>>>>> it does not exists.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.sqoop.common.SqoopException: COMMON_0000:Unable to run
>>>>>>>> specified query - CREATE TABLE "SQOOP"."SQ_INPUT" ("SQI_ID" BIGINT
>>>>>>>> GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY
>>>>>>>> KEY,
>>>>>>>> "SQI_NAME" VARCHAR(64), "SQI_FORM" BIGINT, "SQI_INDEX" SMALLINT,
>>>>>>>> "SQI_TYPE"
>>>>>>>> VARCHAR(32), "SQI_STRMASK" BOOLEAN, "SQI_STRLENGTH" SMALLINT,
>>>>>>>> "SQI_ENUMVALS" VARCHAR(100),CONSTRAINT "SQOOP"."FK_SQI_SQF" FOREIGN KEY
>>>>>>>> ("SQI_FORM") REFERENCES "SQOOP"."SQ_FORM" ("SQF_ID"))
>>>>>>>>
>>>>>>>> Caused by: java.sql.SQLSyntaxErrorException: Syntax error: BOOLEAN.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> This seems to be the root cause. There is “Syntax error: BOOLEAN”
>>>>>>>> when running generate scripts. Would you run this script using “ij” of
>>>>>>>> derby to see whether this script runs correctly in your env?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Richard
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *From:* Lee S [mailto:[email protected]]
>>>>>>>> *Sent:* Tuesday, July 14, 2015 10:56 PM
>>>>>>>> *To:* [email protected]
>>>>>>>> *Subject:* sqoop2-tool verify with exception: The database was
>>>>>>>> created by or upgraded by version 10.8
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi all:
>>>>>>>>
>>>>>>>> I verify the configuration with the exception below:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ERROR XSLAN: Database at
>>>>>>>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db has an incompatible
>>>>>>>> format with the current version of the software. The database was
>>>>>>>> created
>>>>>>>> by or upgraded by version 10.8.
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.iapi.error.StandardException.newException(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> at org.apache.derby.impl.store.raw.RawStore.boot(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> It seems that db in repository is not created with derby version
>>>>>>>> 10.8 , but I dont know how to fix it . I've checked that derby jar in
>>>>>>>> WEB-INF/lib is 10.8.2.2.
>>>>>>>>
>>>>>>>> I'm working with sqoop-1.99.6, any idea?
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
