yep, I compiled it and found that no security classes were available. How about I jumped the compilation of security related module. Cause I need to use sqoop2 in a hadoop2.3 cluster.
On Thu, Jul 16, 2015 at 9:29 AM, Abraham Elmahrek <[email protected]> wrote: > At the moment, Hadoop 2.6 or newer is needed. The reason is that there's a > dependency on some of the security classes that were available in later > versions of Hadoop. > > On Wed, Jul 15, 2015 at 6:26 PM, Lee S <[email protected]> wrote: > >> Hi Abe: >> I'll try what you say. >> And one more question whether sqoop2 can be compiled with hadoop2.3 ? >> I've tried it but seemts that one class can not be found in hadoop2.3. >> >> On Thu, Jul 16, 2015 at 4:18 AM, Abraham Elmahrek <[email protected]> >> wrote: >> >>> So this is a bit strange. There must have been jar conflicts when you >>> were using sqoop2 before. I see a couple of solutions immediately: >>> >>> 1. Start fresh and recreate your jobs >>> 2. Dump the data in the old data base and load into a new database >>> >>> >>> For option two, here's a quick over view of how to do that: >>> >>> 1. Get the DDL with DBLOOK ( >>> http://db.apache.org/derby/docs/10.1/tools/rtoolsdblookexamples.html). >>> To get DBLook working on my mac, I had to do the following: alias >>> dblook='java -cp >>> >>> /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derbytools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/db/lib/derby.jar >>> org.apache.derby.tools.dblook'. Then dblook -d <jdbc conn string>. >>> 2. Get a data dump using SYSCS_UTIL.SYSCS_EXPORT_TABLE system >>> procedure from IJ or programmatically ( >>> http://db.apache.org/derby/docs/10.7/ref/rrefexportproc.html). >>> 3. Load data using SYSCS_UTIL.SYSCS_IMPORT_DATA system procedure >>> from IJ or programmatically ( >>> http://db.apache.org/derby/docs/10.7/ref/rrefimportdataproc.html). >>> >>> Here's a quick example of what a programmatic solution might look like >>> (incomplete): >>> >>> public class DerbyDump { >>>> private static final String SCHEMA_NAME = "SQOOP"; >>>> >>>> private static List<String> fetchTableNames(String jdbc) throws >>>> SQLException { >>>> Connection conn = DriverManager.getConnection(jdbc); >>>> List<String> tableNames = new LinkedList<String>(); >>>> Statement stmt = null; >>>> >>>> try { >>>> stmt = conn.createStatement(); >>>> ResultSet rs = stmt.executeQuery("select t.tablename\n" + >>>> " from sys.systables t, sys.sysschemas s\n" + >>>> " where t.schemaid = s.schemaid\n" + >>>> " and t.tabletype = 'T'\n" + >>>> " and s.schemaname = '" + SCHEMA_NAME + "'\n" + >>>> " order by s.schemaname, t.tablename"); >>>> while (rs.next()) { >>>> tableNames.add(rs.getString(1)); >>>> } >>>> } finally { >>>> if (stmt != null) { >>>> stmt.close(); >>>> } >>>> >>>> if (conn != null) { >>>> conn.close(); >>>> } >>>> } >>>> >>>> return tableNames; >>>> } >>>> >>>> private static void dumpTableToFile(String jdbc, String tableName, >>>> String path) throws SQLException { >>>> Connection conn = DriverManager.getConnection(jdbc); >>>> PreparedStatement stmt = null; >>>> >>>> try { >>>> stmt = conn.prepareStatement("CALL >>>> SYSCS_UTIL.SYSCS_EXPORT_TABLE(?, ?, ?, null, null, 'UTF-8')"); >>>> stmt.setString(1, SCHEMA_NAME); >>>> stmt.setString(2, tableName); >>>> stmt.setString(3, path); >>>> stmt.execute(); >>>> } finally { >>>> if (stmt != null) { >>>> stmt.close(); >>>> } >>>> >>>> if (conn != null) { >>>> conn.close(); >>>> } >>>> } >>>> } >>>> >>>> private static void loadTableToFile(String jdbc, String tableName, >>>> String path) throws SQLException { >>>> Connection conn = DriverManager.getConnection(jdbc); >>>> PreparedStatement stmt = null; >>>> >>>> try { >>>> stmt = conn.prepareStatement("CALL >>>> SYSCS_UTIL.SYSCS_IMPORT_DATA(?, ?, null, null, ?, null, null, 'UTF-8', >>>> 0)"); >>>> stmt.setString(1, SCHEMA_NAME); >>>> stmt.setString(2, tableName); >>>> stmt.setString(3, path); >>>> stmt.execute(); >>>> } finally { >>>> if (stmt != null) { >>>> stmt.close(); >>>> } >>>> >>>> if (conn != null) { >>>> conn.close(); >>>> } >>>> } >>>> } >>>> } >>>> >>> >>> The above snippet would need to be compiled to read from 10.8 databases >>> when dumping data and 10.8.2.2 when loading data. >>> >>> -Abe >>> >>> On Wed, Jul 15, 2015 at 2:54 AM, Lee S <[email protected]> wrote: >>> >>>> Hi Richard: >>>> I cant run ij and dont know where derby is installed. >>>> And I try to reconfigure sqoop and ran it without the exceptions I >>>> posted earlier. >>>> but with the exceptions I sent on the first email. >>>> >>>> >>>> On Wed, Jul 15, 2015 at 10:16 AM, Zhou, Richard <[email protected] >>>> > wrote: >>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> There is no need to install derby in advance. >>>>> >>>>> For the sqoop.log, >>>>> >>>>> >>>>> >>>>> [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.detectRepositoryVersion(DerbyRepositoryHandler.java:196)] >>>>> Can't fetch repository structure version. >>>>> >>>>> Caused by: java.sql.SQLException: Schema 'SQOOP' does not exist >>>>> >>>>> >>>>> >>>>> This error is correct. It’s fine. As it’s your first time to start a >>>>> Sqoop server, it will generate DB automatically (also log this error) if >>>>> it >>>>> does not exists. >>>>> >>>>> >>>>> >>>>> org.apache.sqoop.common.SqoopException: COMMON_0000:Unable to run >>>>> specified query - CREATE TABLE "SQOOP"."SQ_INPUT" ("SQI_ID" BIGINT >>>>> GENERATED ALWAYS AS IDENTITY (START WITH 1, INCREMENT BY 1) PRIMARY KEY, >>>>> "SQI_NAME" VARCHAR(64), "SQI_FORM" BIGINT, "SQI_INDEX" SMALLINT, >>>>> "SQI_TYPE" >>>>> VARCHAR(32), "SQI_STRMASK" BOOLEAN, "SQI_STRLENGTH" SMALLINT, >>>>> "SQI_ENUMVALS" VARCHAR(100),CONSTRAINT "SQOOP"."FK_SQI_SQF" FOREIGN KEY >>>>> ("SQI_FORM") REFERENCES "SQOOP"."SQ_FORM" ("SQF_ID")) >>>>> >>>>> Caused by: java.sql.SQLSyntaxErrorException: Syntax error: BOOLEAN. >>>>> >>>>> >>>>> >>>>> This seems to be the root cause. There is “Syntax error: BOOLEAN” when >>>>> running generate scripts. Would you run this script using “ij” of derby to >>>>> see whether this script runs correctly in your env? >>>>> >>>>> >>>>> >>>>> Regards >>>>> >>>>> Richard >>>>> >>>>> >>>>> >>>>> *From:* Lee S [mailto:[email protected]] >>>>> *Sent:* Tuesday, July 14, 2015 10:56 PM >>>>> *To:* [email protected] >>>>> *Subject:* sqoop2-tool verify with exception: The database was >>>>> created by or upgraded by version 10.8 >>>>> >>>>> >>>>> >>>>> Hi all: >>>>> >>>>> I verify the configuration with the exception below: >>>>> >>>>> >>>>> >>>>> ERROR XSLAN: Database at >>>>> /root/lcy/sqoop-1.99.6-bin-hadoop200/repository/db has an incompatible >>>>> format with the current version of the software. The database was created >>>>> by or upgraded by version 10.8. >>>>> >>>>> at >>>>> org.apache.derby.iapi.error.StandardException.newException(Unknown Source) >>>>> >>>>> at >>>>> org.apache.derby.impl.store.raw.log.LogToFile.readControlFile(Unknown >>>>> Source) >>>>> >>>>> at org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown >>>>> Source) >>>>> >>>>> at >>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) >>>>> >>>>> at >>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>> Source) >>>>> >>>>> at >>>>> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown >>>>> Source) >>>>> >>>>> at >>>>> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown >>>>> Source) >>>>> >>>>> at >>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown >>>>> Source) >>>>> >>>>> at >>>>> org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown >>>>> Source) >>>>> >>>>> at org.apache.derby.impl.store.raw.RawStore.boot(Unknown >>>>> Source) >>>>> >>>>> at >>>>> org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) >>>>> >>>>> at >>>>> org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown >>>>> Source) >>>>> >>>>> >>>>> >>>>> It seems that db in repository is not created with derby version 10.8 >>>>> , but I dont know how to fix it . I've checked that derby jar in >>>>> WEB-INF/lib is 10.8.2.2. >>>>> >>>>> I'm working with sqoop-1.99.6, any idea? >>>>> >>>> >>>> >>> >> >
