Got it. Thanks for your clear answers. On Thu, Jul 16, 2015 at 9:39 AM, Abraham Elmahrek <[email protected]> wrote:
> Sqoop2 should work against Hadoop 2.3 if it's compiled against Hadoop 2.6. > Those versions are wire compatible. If you run into any issues, post them > here and we can see exactly what's going on. > > > http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-common/Compatibility.html#Wire_compatibility > > On Wed, Jul 15, 2015 at 6:35 PM, Lee S <[email protected]> wrote: > >> yep, I compiled it and found that no security classes were available. >> How about I jumped the compilation of security related module. >> Cause I need to use sqoop2 in a hadoop2.3 cluster. >> >> On Thu, Jul 16, 2015 at 9:29 AM, Abraham Elmahrek <[email protected]> >> wrote: >> >>> At the moment, Hadoop 2.6 or newer is needed. The reason is that there's >>> a dependency on some of the security classes that were available in later >>> versions of Hadoop. >>> >>> On Wed, Jul 15, 2015 at 6:26 PM, Lee S <[email protected]> wrote: >>> >>>> Hi Abe: >>>> I'll try what you say. >>>> And one more question whether sqoop2 can be compiled with hadoop2.3 ? >>>> I've tried it but seemts that one class can not be found in hadoop2.3. >>>> >>>> On Thu, Jul 16, 2015 at 4:18 AM, Abraham Elmahrek <[email protected]> >>>> wrote: >>>> >>>>> So this is a bit strange. There must have been jar conflicts when you >>>>> were using sqoop2 before. I see a couple of solutions immediately: >>>>> >>>>> 1. Start fresh and recreate your jobs >>>>> 2. Dump the data in the old data base and load into a new database >>>>> >>>>> >>>>> For option two, here's a quick over view of how to do that: >>>>> >>>>> 1. Get the DDL with DBLOOK ( >>>>> http://db.apache.org/derby/docs/10.1/tools/rtoolsdblookexamples.html). >>>>> To get DBLook working on my mac, I had to do the following: alias >>>>> dblook='java -cp >>>>> >>>>> /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derbytools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derby.jar >>>>> org.apache.derby.tools.dblook'. Then dblook -d <jdbc conn string>. >>>>> 2. Get a data dump using SYSCS_UTIL.SYSCS_EXPORT_TABLE system >>>>> procedure from IJ or programmatically ( >>>>> http://db.apache.org/derby/docs/10.7/ref/rrefexportproc.html). >>>>> 3. Load data using SYSCS_UTIL.SYSCS_IMPORT_DATA system procedure >>>>> from IJ or programmatically ( >>>>> http://db.apache.org/derby/docs/10.7/ref/rrefimportdataproc.html). >>>>> >>>>> Here's a quick example of what a programmatic solution might look like >>>>> (incomplete): >>>>> >>>>> public class DerbyDump { >>>>>> private static final String SCHEMA_NAME = "SQOOP"; >>>>>> >>>>>> private static List<String> fetchTableNames(String jdbc) throws >>>>>> SQLException { >>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>> List<String> tableNames = new LinkedList<String>(); >>>>>> Statement stmt = null; >>>>>> >>>>>> try { >>>>>> stmt = conn.createStatement(); >>>>>> ResultSet rs = stmt.executeQuery("select t.tablename\n" + >>>>>> " from sys.systables t, sys.sysschemas s\n" + >>>>>> " where t.schemaid = s.schemaid\n" + >>>>>> " and t.tabletype = 'T'\n" + >>>>>> " and s.schemaname = '" + SCHEMA_NAME + "'\n" + >>>>>> " order by s.schemaname, t.tablename"); >>>>>> while (rs.next()) { >>>>>> tableNames.add(rs.getString(1)); >>>>>> } >>>>>> } finally { >>>>>> if (stmt != null) { >>>>>> stmt.close(); >>>>>> } >>>>>> >>>>>> if (conn != null) { >>>>>> conn.close(); >>>>>> } >>>>>> } >>>>>> >>>>>> return tableNames; >>>>>> } >>>>>> >>>>>> private static void dumpTableToFile(String jdbc, String tableName, >>>>>> String path) throws SQLException { >>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>> PreparedStatement stmt = null; >>>>>> >>>>>> try { >>>>>> stmt = conn.prepareStatement("CALL >>>>>> SYSCS_UTIL.SYSCS_EXPORT_TABLE(?, ?, ?, null, null, 'UTF-8')"); >>>>>> stmt.setString(1, SCHEMA_NAME); >>>>>> stmt.setString(2, tableName); >>>>>> stmt.setString(3, path); >>>>>> stmt.execute(); >>>>>> } finally { >>>>>> if (stmt != null) { >>>>>> stmt.close(); >>>>>> } >>>>>> >>>>>> if (conn != null) { >>>>>> conn.close(); >>>>>> } >>>>>> } >>>>>> } >>>>>> >>>>>> private static void loadTableToFile(String jdbc, String tableName, >>>>>> String path) throws SQLException { >>>>>> Connection conn = DriverManager.getConnection(jdbc); >>>>>> PreparedStatement stmt = null; >>>>>> >>>>>> try { >>>>>> stmt = conn.prepareStatement("CALL >>>>>> SYSCS_UTIL.SYSCS_IMPORT_DATA(?, ?, null, null, ?, null, null, 'UTF-8', >>>>>> 0)"); >>>>>> stmt.setString(1, SCHEMA_NAME); >>>>>> stmt.setString(2, tableName); >>>>>> stmt.setString(3, path); >>>>>> stmt.execute(); >>>>>> } finally { >>>>>> if (stmt != null) { >>>>>> stmt.close(); >>>>>> } >>>>>> >>>>>> if (conn != null) { >>>>>> conn.close(); >>>>>> } >>>>>> } >>>>>> } >>>>>> } >>>>>> >>>>> >>>>> The above snippet would need to be compiled to read from 10.8 >>>>> databases when dumping data and 10.8.2.2 when loading data. >>>>> >>>>> -Abe >>>>> >>>>> On Wed, Jul 15, 2015 at 2:54 AM, Lee S <[email protected]> wrote: >>>>> >>>>>> Hi Richard: >>>>>> I cant run ij and dont know where derby is installed. >>>>>> And I try to reconfigure sqoop and ran it without the exceptions I >>>>>> posted earlier. >>>>>> but with the exceptions I sent on the first email. >>>>>> >>>>>> >>>>>> On Wed, Jul 15, 2015 at 10:16 AM, Zhou, Richard < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> There is no need to install derby in advance. >>>>>>> >>>>>>> For the sqoop.log, >>>>>>> >>>>>>> >>>>>>> >>>>>>> [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.detectRepositoryVersion(DerbyRepositoryHandler.java:196)] >>>>>>> Can't fetch repository structure version. >>>>>>> >>>>>>> Caused by: java.sql.SQLException: Schema 'SQOOP' does not exist >>>>>>> >>>>>>> >>>>>>> >>>>>>> This error is correct. It’s fine. As it’s your first time to start a >>>>>>> Sqoop server, it will generate DB automatically (also log this error) >>>>>>> if it >>>>>>> does not exists. >>>>>>> >>>>>>> >>>>>>> >>>>>>> org.apache.sqoop.common.SqoopException: COMMON_0000:Unable to run >>>>>>> specified query - CREATE TABLE "SQOOP"."SQ_INPUT" ("SQI_ID" BIGINT >>>>>>> GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY KEY, >>>>>>> "SQI_NAME" VARCHAR(64), "SQI_FORM" BIGINT, "SQI_INDEX" SMALLINT, >>>>>>> "SQI_TYPE" >>>>>>> VARCHAR(32), "SQI_STRMASK" BOOLEAN, "SQI_STRLENGTH" SMALLINT, >>>>>>> "SQI_ENUMVALS" VARCHAR(100),CONSTRAINT "SQOOP"."FK_SQI_SQF" FOREIGN KEY >>>>>>> ("SQI_FORM") REFERENCES "SQOOP"."SQ_FORM" ("SQF_ID")) >>>>>>> >>>>>>> Caused by: java.sql.SQLSyntaxErrorException: Syntax error: BOOLEAN. >>>>>>> >>>>>>> >>>>>>> >>>>>>> This seems to be the root cause. There is “Syntax error: BOOLEAN” >>>>>>> when running generate scripts. Would you run this script using “ij” of >>>>>>> derby to see whether this script runs correctly in your env? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> Richard >>>>>>> >>>>>>> >>>>>>> >>>>>>> *From:* Lee S [mailto:[email protected]] >>>>>>> *Sent:* Tuesday, July 14, 2015 10:56 PM >>>>>>> *To:* [email protected] >>>>>>> *Subject:* sqoop2-tool verify with exception: The database was >>>>>>> created by or upgraded by version 10.8 >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi all: >>>>>>> >>>>>>> I verify the configuration with the exception below: >>>>>>> >>>>>>> >>>>>>> >>>>>>> ERROR XSLAN: Database at >>>>>>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db has an incompatible >>>>>>> format with the current version of the software. The database was >>>>>>> created >>>>>>> by or upgraded by version 10.8. >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.iapi.error.StandardException.newException(Unknown >>>>>>> Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown >>>>>>> Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>>> Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown >>>>>>> Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown >>>>>>> Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown >>>>>>> Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown >>>>>>> Source) >>>>>>> >>>>>>> at org.apache.derby.impl.store.raw.RawStore.boot(Unknown >>>>>>> Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>>>> Source) >>>>>>> >>>>>>> >>>>>>> >>>>>>> It seems that db in repository is not created with derby version >>>>>>> 10.8 , but I dont know how to fix it . I've checked that derby jar in >>>>>>> WEB-INF/lib is 10.8.2.2. >>>>>>> >>>>>>> I'm working with sqoop-1.99.6, any idea? >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
