Sqoop2 should work against Hadoop 2.3 if it's compiled against Hadoop 2.6. Those versions are wire compatible. If you run into any issues, post them here and we can see exactly what's going on.
http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility On Wed, Jul 15, 2015 at 6:35 PM, Lee S <[email protected]> wrote: > yep, I compiled it and found that no security classes were available. > How about I jumped the compilation of security related module. > Cause I need to use sqoop2 in a hadoop2.3 cluster. > > On Thu, Jul 16, 2015 at 9:29 AM, Abraham Elmahrek <[email protected]> > wrote: > >> At the moment, Hadoop 2.6 or newer is needed. The reason is that there's >> a dependency on some of the security classes that were available in later >> versions of Hadoop. >> >> On Wed, Jul 15, 2015 at 6:26 PM, Lee S <[email protected]> wrote: >> >>> Hi Abe: >>> I'll try what you say. >>> And one more question whether sqoop2 can be compiled with hadoop2.3 ? >>> I've tried it but seemts that one class can not be found in hadoop2.3. >>> >>> On Thu, Jul 16, 2015 at 4:18 AM, Abraham Elmahrek <[email protected]> >>> wrote: >>> >>>> So this is a bit strange. There must have been jar conflicts when you >>>> were using sqoop2 before. I see a couple of solutions immediately: >>>> >>>> 1. Start fresh and recreate your jobs >>>> 2. Dump the data in the old data base and load into a new database >>>> >>>> >>>> For option two, here's a quick over view of how to do that: >>>> >>>> 1. Get the DDL with DBLOOK ( >>>> http://db.apache.org/derby/docs/10.1/tools/rtoolsdblookexamples.html). >>>> To get DBLook working on my mac, I had to do the following: alias >>>> dblook='java -cp >>>> >>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derbytools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derby.jar >>>> org.apache.derby.tools.dblook'. Then dblook -d <jdbc conn string>. >>>> 2. Get a data dump using SYSCS_UTIL.SYSCS_EXPORT_TABLE system >>>> procedure from IJ or programmatically ( >>>> http://db.apache.org/derby/docs/10.7/ref/rrefexportproc.html). >>>> 3. Load data using SYSCS_UTIL.SYSCS_IMPORT_DATA system procedure >>>> from IJ or programmatically ( >>>> http://db.apache.org/derby/docs/10.7/ref/rrefimportdataproc.html). >>>> >>>> Here's a quick example of what a programmatic solution might look like >>>> (incomplete): >>>> >>>> public class DerbyDump { >>>>> private static final String SCHEMA_NAME = "SQOOP"; >>>>> >>>>> private static List<String> fetchTableNames(String jdbc) throws >>>>> SQLException { >>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>> List<String> tableNames = new LinkedList<String>(); >>>>> Statement stmt = null; >>>>> >>>>> try { >>>>> stmt = conn.createStatement(); >>>>> ResultSet rs = stmt.executeQuery("select t.tablename\n" + >>>>> " from sys.systables t, sys.sysschemas s\n" + >>>>> " where t.schemaid = s.schemaid\n" + >>>>> " and t.tabletype = 'T'\n" + >>>>> " and s.schemaname = '" + SCHEMA_NAME + "'\n" + >>>>> " order by s.schemaname, t.tablename"); >>>>> while (rs.next()) { >>>>> tableNames.add(rs.getString(1)); >>>>> } >>>>> } finally { >>>>> if (stmt != null) { >>>>> stmt.close(); >>>>> } >>>>> >>>>> if (conn != null) { >>>>> conn.close(); >>>>> } >>>>> } >>>>> >>>>> return tableNames; >>>>> } >>>>> >>>>> private static void dumpTableToFile(String jdbc, String tableName, >>>>> String path) throws SQLException { >>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>> PreparedStatement stmt = null; >>>>> >>>>> try { >>>>> stmt = conn.prepareStatement("CALL >>>>> SYSCS_UTIL.SYSCS_EXPORT_TABLE(?, ?, ?, null, null, 'UTF-8')"); >>>>> stmt.setString(1, SCHEMA_NAME); >>>>> stmt.setString(2, tableName); >>>>> stmt.setString(3, path); >>>>> stmt.execute(); >>>>> } finally { >>>>> if (stmt != null) { >>>>> stmt.close(); >>>>> } >>>>> >>>>> if (conn != null) { >>>>> conn.close(); >>>>> } >>>>> } >>>>> } >>>>> >>>>> private static void loadTableToFile(String jdbc, String tableName, >>>>> String path) throws SQLException { >>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>> PreparedStatement stmt = null; >>>>> >>>>> try { >>>>> stmt = conn.prepareStatement("CALL >>>>> SYSCS_UTIL.SYSCS_IMPORT_DATA(?, ?, null, null, ?, null, null, 'UTF-8', >>>>> 0)"); >>>>> stmt.setString(1, SCHEMA_NAME); >>>>> stmt.setString(2, tableName); >>>>> stmt.setString(3, path); >>>>> stmt.execute(); >>>>> } finally { >>>>> if (stmt != null) { >>>>> stmt.close(); >>>>> } >>>>> >>>>> if (conn != null) { >>>>> conn.close(); >>>>> } >>>>> } >>>>> } >>>>> } >>>>> >>>> >>>> The above snippet would need to be compiled to read from 10.8 databases >>>> when dumping data and 10.8.2.2 when loading data. >>>> >>>> -Abe >>>> >>>> On Wed, Jul 15, 2015 at 2:54 AM, Lee S <[email protected]> wrote: >>>> >>>>> Hi Richard: >>>>> I cant run ij and dont know where derby is installed. >>>>> And I try to reconfigure sqoop and ran it without the exceptions I >>>>> posted earlier. >>>>> but with the exceptions I sent on the first email. >>>>> >>>>> >>>>> On Wed, Jul 15, 2015 at 10:16 AM, Zhou, Richard < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> There is no need to install derby in advance. >>>>>> >>>>>> For the sqoop.log, >>>>>> >>>>>> >>>>>> >>>>>> [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.detectRepositoryVersion(DerbyRepositoryHandler.java:196)] >>>>>> Can't fetch repository structure version. >>>>>> >>>>>> Caused by: java.sql.SQLException: Schema 'SQOOP' does not exist >>>>>> >>>>>> >>>>>> >>>>>> This error is correct. It’s fine. As it’s your first time to start a >>>>>> Sqoop server, it will generate DB automatically (also log this error) if >>>>>> it >>>>>> does not exists. >>>>>> >>>>>> >>>>>> >>>>>> org.apache.sqoop.common.SqoopException: COMMON_0000:Unable to run >>>>>> specified query - CREATE TABLE "SQOOP"."SQ_INPUT" ("SQI_ID" BIGINT >>>>>> GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY KEY, >>>>>> "SQI_NAME" VARCHAR(64), "SQI_FORM" BIGINT, "SQI_INDEX" SMALLINT, >>>>>> "SQI_TYPE" >>>>>> VARCHAR(32), "SQI_STRMASK" BOOLEAN, "SQI_STRLENGTH" SMALLINT, >>>>>> "SQI_ENUMVALS" VARCHAR(100),CONSTRAINT "SQOOP"."FK_SQI_SQF" FOREIGN KEY >>>>>> ("SQI_FORM") REFERENCES "SQOOP"."SQ_FORM" ("SQF_ID")) >>>>>> >>>>>> Caused by: java.sql.SQLSyntaxErrorException: Syntax error: BOOLEAN. >>>>>> >>>>>> >>>>>> >>>>>> This seems to be the root cause. There is “Syntax error: BOOLEAN” >>>>>> when running generate scripts. Would you run this script using “ij” of >>>>>> derby to see whether this script runs correctly in your env? >>>>>> >>>>>> >>>>>> >>>>>> Regards >>>>>> >>>>>> Richard >>>>>> >>>>>> >>>>>> >>>>>> *From:* Lee S [mailto:[email protected]] >>>>>> *Sent:* Tuesday, July 14, 2015 10:56 PM >>>>>> *To:* [email protected] >>>>>> *Subject:* sqoop2-tool verify with exception: The database was >>>>>> created by or upgraded by version 10.8 >>>>>> >>>>>> >>>>>> >>>>>> Hi all: >>>>>> >>>>>> I verify the configuration with the exception below: >>>>>> >>>>>> >>>>>> >>>>>> ERROR XSLAN: Database at >>>>>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db has an incompatible >>>>>> format with the current version of the software. The database was >>>>>> created >>>>>> by or upgraded by version 10.8. >>>>>> >>>>>> at >>>>>> org.apache.derby.iapi.error.StandardException.newException(Unknown >>>>>> Source) >>>>>> >>>>>> at >>>>>> org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown >>>>>> Source) >>>>>> >>>>>> at org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown >>>>>> Source) >>>>>> >>>>>> at >>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) >>>>>> >>>>>> at >>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>> Source) >>>>>> >>>>>> at >>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown >>>>>> Source) >>>>>> >>>>>> at >>>>>> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown >>>>>> Source) >>>>>> >>>>>> at >>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown >>>>>> Source) >>>>>> >>>>>> at >>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown >>>>>> Source) >>>>>> >>>>>> at org.apache.derby.impl.store.raw.RawStore.boot(Unknown >>>>>> Source) >>>>>> >>>>>> at >>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) >>>>>> >>>>>> at >>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>> Source) >>>>>> >>>>>> >>>>>> >>>>>> It seems that db in repository is not created with derby version 10.8 >>>>>> , but I dont know how to fix it . I've checked that derby jar in >>>>>> WEB-INF/lib is 10.8.2.2. >>>>>> >>>>>> I'm working with sqoop-1.99.6, any idea? >>>>>> >>>>> >>>>> >>>> >>> >> >
