ok, thanks Abe . I will try it .
On Thu Nov 20 2014 at 上午10:55:08 Abraham Elmahrek <[email protected]> wrote: > https://issues.apache.org/jira/browse/SQOOP-1168 > > We're shooting for 1.99.5: > https://cwiki.apache.org/confluence/display/SQOOP/Sqoop+2+Roadmap. Sqoop2 > development is pretty active these days. Hopefully in the next few months > we'll see 1.99.5. 1.99.4 is in the process of being released. > > Until then though, if you just need incremental import, try using an > apache bigtop distribution of Sqoop: http://bigtop.apache.org/, > http://mirror.cogentco.com/pub/apache/bigtop/bigtop-0.8.0/repos/. It will > give you packages to install using your favorite flavor of linux. > > If the above doesn't work, you can download binary distributions of Sqoop: > http://www.apache.org/dyn/closer.cgi/sqoop/1.4.5. Choose a mirror and > download the distribution marked hadoop-2.0.4-alpha. > > If that doesn't work either, you can try one of the many vendors that > provide Hadoop. > > -Abe > > On Wed, Nov 19, 2014 at 6:45 PM, 呂佳鴻 <[email protected]> wrote: > >> I need to use the incremental import of sqoop. >> But i noticed that sqoop2 can't support it ? >> Or it can support incremental import ? >> >> Thank you >> >> Luke >> >> >> >> >> On Thu Nov 20 2014 at 上午10:36:37 Abraham Elmahrek <[email protected]> >> wrote: >> >>> Building against a later version of Hadoop is highly discouraged. >>> Certain client jars have changed which will probably cause issues (likely >>> this one). >>> >>> What's your use case? Maybe we can figure something else out? >>> >>> Also, checkout Sqoop2 for a better Java client if that's what you need. >>> >>> NOTE: Hadoop maintains wire compatibility for major versions. This means >>> that old clients can be used against newer servers as long as they share >>> the same major version number. >>> >>> On Tue, Nov 18, 2014 at 9:27 PM, 呂佳鴻 <[email protected]> wrote: >>> >>>> Hi, i try not to change build.xml now , and still have the same >>>> problem. >>>> (just use "ant package" to build) >>>> >>>> I modified the sqoop.env.sh to add HADOOP_COMMON_HOME & >>>> HADOOP_MAPRED_HOME >>>> >>>> Then I add mysql-connector-java-5.1.34-bin.jar to sqoop lib >>>> try the same command >>>> >>>> sqoop import --connect jdbc:mysql://localhost:3306/test --username xxx >>>> --password xxxx --table user1 -m 1 >>>> >>>> How can I modified the error? >>>> so I can run sqoop on hadoop 2.4.1? >>>> if there is other solution or file for hadoop 2.4.1 that i can check . >>>> >>>> java.lang.NoSuchMethodError: >>>> org.apache.hadoop.http.HttpConfig.getSchemePrefix()Ljava/lang/String; >>>> >>>> >>>> Thank you! >>>> >>>> Luke >>>> >>>> >>>> >>>> >>>> ---------------------------- >>>> follow are appear message >>>> Warning: /opt/sqoop-1.4.5/../hcatalog does not exist! HCatalog jobs >>>> will fail. >>>> Please set $HCAT_HOME to the root of your HCatalog installation. >>>> Warning: /opt/sqoop-1.4.5/../accumulo does not exist! Accumulo imports >>>> will fail. >>>> Please set $ACCUMULO_HOME to the root of your Accumulo installation. >>>> Warning: /opt/sqoop-1.4.5/../zookeeper does not exist! Accumulo imports >>>> will fail. >>>> Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. >>>> 14/11/19 13:21:47 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5 >>>> 14/11/19 13:21:47 WARN tool.BaseSqoopTool: Setting your password on the >>>> command-line is insecure. Consider using -P instead. >>>> 14/11/19 13:21:48 INFO manager.MySQLManager: Preparing to use a MySQL >>>> streaming resultset. >>>> 14/11/19 13:21:48 INFO tool.CodeGenTool: Beginning code generation >>>> SLF4J: Class path contains multiple SLF4J bindings. >>>> SLF4J: Found binding in >>>> [jar:file:/opt/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>>> SLF4J: Found binding in >>>> [jar:file:/opt/hbase-0.98.4/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>>> SLF4J: Found binding in >>>> [jar:file:/opt/sqoop-1.4.5/build/ivy/lib/sqoop/hadoop200/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>>> explanation. >>>> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>>> 14/11/19 13:21:48 INFO manager.SqlManager: Executing SQL statement: >>>> SELECT t.* FROM `user1` AS t LIMIT 1 >>>> 14/11/19 13:21:48 INFO manager.SqlManager: Executing SQL statement: >>>> SELECT t.* FROM `user1` AS t LIMIT 1 >>>> 14/11/19 13:21:48 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is >>>> /opt/hadoop-2.4.1/share/hadoop/mapreduce >>>> Note: >>>> /tmp/sqoop-hadoop/compile/4e64e5e221ace9afdb32576604ed074b/user1.java uses >>>> or overrides a deprecated API. >>>> Note: Recompile with -Xlint:deprecation for details. >>>> 14/11/19 13:21:49 INFO orm.CompilationManager: Writing jar file: >>>> /tmp/sqoop-hadoop/compile/4e64e5e221ace9afdb32576604ed074b/user1.jar >>>> 14/11/19 13:21:50 WARN manager.MySQLManager: It looks like you are >>>> importing from mysql. >>>> 14/11/19 13:21:50 WARN manager.MySQLManager: This transfer can be >>>> faster! Use the --direct >>>> 14/11/19 13:21:50 WARN manager.MySQLManager: option to exercise a >>>> MySQL-specific fast path. >>>> 14/11/19 13:21:50 INFO manager.MySQLManager: Setting zero DATETIME >>>> behavior to convertToNull (mysql) >>>> 14/11/19 13:21:50 INFO mapreduce.ImportJobBase: Beginning import of >>>> user1 >>>> Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library >>>> /opt/hadoop-2.4.1/lib/native/libhadoop.so.1.0.0 which might have disabled >>>> stack guard. The VM will try to fix the stack guard now. >>>> It's highly recommended that you fix the library with 'execstack -c >>>> <libfile>', or link it with '-z noexecstack'. >>>> 14/11/19 13:21:50 WARN util.NativeCodeLoader: Unable to load >>>> native-hadoop library for your platform... using builtin-java classes where >>>> applicable >>>> 14/11/19 13:21:50 INFO Configuration.deprecation: mapred.jar is >>>> deprecated. Instead, use mapreduce.job.jar >>>> 14/11/19 13:21:50 INFO Configuration.deprecation: mapred.map.tasks is >>>> deprecated. Instead, use mapreduce.job.maps >>>> 14/11/19 13:21:50 INFO client.RMProxy: Connecting to ResourceManager at >>>> /0.0.0.0:8032 >>>> 14/11/19 13:21:52 INFO db.DBInputFormat: Using read commited >>>> transaction isolation >>>> 14/11/19 13:21:52 INFO mapreduce.JobSubmitter: number of splits:1 >>>> 14/11/19 13:21:52 INFO mapreduce.JobSubmitter: Submitting tokens for >>>> job: job_1416370328669_0005 >>>> 14/11/19 13:21:52 INFO impl.YarnClientImpl: Submitted application >>>> application_1416370328669_0005 >>>> 14/11/19 13:21:52 INFO mapreduce.JobSubmitter: Cleaning up the staging >>>> area /tmp/hadoop-yarn/staging/hadoop/.staging/job_1416370328669_0005 >>>> Exception in thread "main" java.lang.NoSuchMethodError: >>>> org.apache.hadoop.http.HttpConfig.getSchemePrefix()Ljava/lang/String; >>>> at >>>> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:428) >>>> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:302) >>>> at >>>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430) >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) >>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) >>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) >>>> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286) >>>> at >>>> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186) >>>> at >>>> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159) >>>> at >>>> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:247) >>>> at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:665) >>>> at >>>> org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118) >>>> at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) >>>> at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601) >>>> at org.apache.sqoop.Sqoop.run(Sqoop.java:143) >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) >>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) >>>> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) >>>> at org.apache.sqoop.Sqoop.main(Sqoop.java:236) >>>> On Tue Nov 18 2014 at 上午1:38:33 Abraham Elmahrek <[email protected]> >>>> wrote: >>>> >>>>> You shouldn't need to change the versions in build.xml. If should just >>>>> work out of the box with any Hadoop 2 distribution. >>>>> >>>>> -Abe >>>>> >>>>> On Mon, Nov 17, 2014 at 2:33 AM, lu <[email protected]> wrote: >>>>> >>>>>> hi , >>>>>> i download sqoop-1.4.5.tar.gz ,and modified build.xml >>>>>> (just change hadoop version to 2.4.1 ) >>>>>> <equals arg1="${hadoopversion}" arg2="200" /> >>>>>> <then> >>>>>> <property name="hadoop.version" value="2.4.1" /> >>>>>> <property name="hbase94.version" value="0.94.2" /> >>>>>> <property name="zookeeper.version" value="3.4.2" /> >>>>>> <property name="hadoop.version.full" value="2.4.1" /> >>>>>> <property name="hcatalog.version" value="0.13.0" /> >>>>>> <property name="hbasecompatprofile" value="2" /> >>>>>> <property name="avrohadoopprofile" value="2" /> >>>>>> </then> >>>>>> >>>>>> after build and install >>>>>> i try to convert data from mysql to hdfs >>>>>> "sqoop import --connect jdbc:mysql://localhost:3306/test --username >>>>>> xxx --password xxxx --table user1 -m 1 --verbose" >>>>>> it's appear >>>>>> >>>>>> *Exception in thread "main" java.lang.NoSuchMethodError: >>>>>> org.apache.hadoop.http.HttpConfig.getSchemePrefix()Ljava/lang/String;* >>>>>> >>>>>> this method seem to change in hadoop 2.4 >>>>>> how can i fix this bug and if sqoop 1.4.5 can support hadoop 2.4.1? >>>>>> i can't find solution on web >>>>>> >>>>>> >>>>>> Thank you! >>>>>> >>>>>> Luke >>>>>> >>>>> >>>>> >>> >
