Pls Ignore the previous mail Thanks guys for the help and for info found that "--batch" option is working which support batch export without " -Dsqoop.export.records.per.statement=1".
*Regards Muthupandi.K* [image: Picture (Device Independent Bitmap)] On Mon, Aug 25, 2014 at 11:18 AM, Muthu Pandi <muthu1...@gmail.com> wrote: > Thanks guys for the help and for info found that "--direct" option is > working which support batch export without " -Dsqoop.export.records.per. > statement=1". > > > > *Regards Muthupandi.K* > > [image: Picture (Device Independent Bitmap)] > > > > On Mon, Aug 25, 2014 at 11:04 AM, Gwen Shapira <gshap...@cloudera.com> > wrote: > >> It won't be super fast since its row-by-row insert, but 2GB is not huge, >> so I'll expect this will take only few minutes. >> >> >> On Sun, Aug 24, 2014 at 10:23 PM, Muthu Pandi <muthu1...@gmail.com> >> wrote: >> >>> Spot on!!! Jarcec >>> >>> Thanks for the quick reply, you have saved my lot of time. >>> >>> Can this be used for exporting larger data since we are exporting >>> in bulk. My data will be of size 2GB Per day and will export at end of the >>> day. >>> >>> >>> >>> >>> *Regards Muthupandi.K* >>> >>> [image: Picture (Device Independent Bitmap)] >>> >>> >>> >>> On Mon, Aug 25, 2014 at 10:40 AM, Jarek Jarcec Cecho <jar...@apache.org> >>> wrote: >>> >>>> Hi Muthu, >>>> the given message is a “Warning” and not an “Error”. Sqoop is warning >>>> you that you are using —driver option that will force Sqoop to use Generic >>>> JDBC connector. That is fine in your case as Sqoop don’t have specialized >>>> connector for Vertica. You can always use --connect-manager parameter to >>>> get rid of the message as is the warning suggesting: >>>> >>>> sqoop export --connect-manager >>>> org.apache.sqoop.manager.GenericJdbcManager ... >>>> >>>> You are however hitting a different problem, my best guess that Vertica >>>> is not supporting multi value insert statements that Sqoop is generating by >>>> default. Try to disable this feature using property >>>> sqoop.export.records.per.statement: >>>> >>>> sqoop export -Dsqoop.export.records.per.statement=1 —connect ... >>>> >>>> Jarcec >>>> >>>> On Aug 25, 2014, at 6:55 AM, Muthu Pandi <muthu1...@gmail.com> wrote: >>>> >>>> > Hi every one, >>>> > Is it possible to export data from HDFS to vetica.I >>>> successfully imported data from vertica to HDFS using sqoop.When i Export i >>>> get the following error.Please give me solution as soon as possible.this is >>>> the error which am getting, >>>> > >>>> > WARN sqoop.ConnFactory: Parameter --driver is set to an explicit >>>> driver however appropriate connection manager is not being set (via >>>> --connection-manager). Sqoop is going to fall back to >>>> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly >>>> which connection manager should be used next time. >>>> > >>>> > will above will have impact.How do i create connection manager for >>>> vertica. >>>> > >>>> > My sqoop query is >>>> > >>>> > sqoop export --driver com.vertica.jdbc.Driver --connect >>>> "jdbc:vertica://X.X.X.X:5433/dbname" --username username -P --table >>>> tablename -m 1 --export-dir /ver.txt --input-fields-terminated-by ',' >>>> --lines-terminated-by '\n' >>>> > >>>> > >>>> > and am using hadoop2.4.1 and sqoop1.4.4 version >>>> > >>>> > >>>> > >>>> > 14/08/25 10:12:57 WARN tool.BaseSqoopTool: Setting your password on >>>> the command-line is insecure. Consider using -P instead. >>>> > 14/08/25 10:12:57 WARN sqoop.ConnFactory: Parameter --driver is set >>>> to an explicit driver however appropriate connection manager is not being >>>> set (via --connection-manager). Sqoop is going to fall back to >>>> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly >>>> which connection manager should be used next time. >>>> > 14/08/25 10:12:57 INFO manager.SqlManager: Using default fetchSize of >>>> 1000 >>>> > 14/08/25 10:12:57 INFO tool.CodeGenTool: Beginning code generation >>>> > 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement: >>>> SELECT t.* FROM AGE AS t WHERE 1=0 >>>> > 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement: >>>> SELECT t.* FROM AGE AS t WHERE 1=0 >>>> > 14/08/25 10:12:58 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is >>>> /opt/hadoop-2.4.1 >>>> > Note: >>>> /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.java uses >>>> or overrides a deprecated API. >>>> > Note: Recompile with -Xlint:deprecation for details. >>>> > 14/08/25 10:13:02 INFO orm.CompilationManager: Writing jar file: >>>> /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.jar >>>> > 14/08/25 10:13:03 INFO mapreduce.ExportJobBase: Beginning export of >>>> AGE >>>> > SLF4J: Class path contains multiple SLF4J bindings. >>>> > SLF4J: Found binding in >>>> [jar:file:/opt/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>>> > SLF4J: Found binding in >>>> [jar:file:/opt/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] >>>> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an >>>> explanation. >>>> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] >>>> > 14/08/25 10:13:03 INFO Configuration.deprecation: mapred.jar is >>>> deprecated. Instead, use mapreduce.job.jar >>>> > 14/08/25 10:13:06 INFO manager.SqlManager: Executing SQL statement: >>>> SELECT t.* FROM AGE AS t WHERE 1=0 >>>> > 14/08/25 10:13:06 INFO Configuration.deprecation: >>>> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use >>>> mapreduce.reduce.speculative >>>> > 14/08/25 10:13:06 INFO Configuration.deprecation: >>>> mapred.map.tasks.speculative.execution is deprecated. Instead, use >>>> mapreduce.map.speculative >>>> > 14/08/25 10:13:06 INFO Configuration.deprecation: mapred.map.tasks is >>>> deprecated. Instead, use mapreduce.job.maps >>>> > 14/08/25 10:13:06 INFO client.RMProxy: Connecting to ResourceManager >>>> at nn01/10.10.200.74:8032 >>>> > 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to >>>> process : 1 >>>> > 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to >>>> process : 1 >>>> > 14/08/25 10:13:27 INFO mapreduce.JobSubmitter: number of splits:1 >>>> > 14/08/25 10:13:28 INFO mapreduce.JobSubmitter: Submitting tokens for >>>> job: job_1408941947869_0001 >>>> > 14/08/25 10:13:30 INFO impl.YarnClientImpl: Submitted application >>>> application_1408941947869_0001 >>>> > 14/08/25 10:13:30 INFO mapreduce.Job: The url to track the job: >>>> http://nn01:8088/proxy/application_1408941947869_0001/ >>>> > 14/08/25 10:13:30 INFO mapreduce.Job: Running job: >>>> job_1408941947869_0001 >>>> > 14/08/25 10:13:44 INFO mapreduce.Job: Job job_1408941947869_0001 >>>> running in uber mode : false >>>> > 14/08/25 10:13:44 INFO mapreduce.Job: map 0% reduce 0% >>>> > 14/08/25 10:14:11 INFO mapreduce.Job: map 100% reduce 0% >>>> > 14/08/25 10:14:11 INFO mapreduce.Job: Task Id : >>>> attempt_1408941947869_0001_m_000000_0, Status : FAILED >>>> > Error: java.io.IOException: java.sql.SQLException: >>>> [Vertica][VJDBC](4856) ERROR: Syntax error at or near "," >>>> > at >>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184) >>>> > at >>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647) >>>> > at >>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770) >>>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) >>>> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) >>>> > at java.security.AccessController.doPrivileged(Native Method) >>>> > at javax.security.auth.Subject.doAs(Subject.java:415) >>>> > at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) >>>> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) >>>> > Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: >>>> Syntax error at or near "," >>>> > at com.vertica.util.ServerErrorData.buildException(Unknown >>>> Source) >>>> > at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown >>>> Source) >>>> > at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown >>>> Source) >>>> > at com.vertica.dataengine.VDataEngine.prepare(Unknown Source) >>>> > at com.vertica.dataengine.VDataEngine.prepare(Unknown Source) >>>> > at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source) >>>> > at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown >>>> Source) >>>> > at >>>> com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source) >>>> > at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source) >>>> > at >>>> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141) >>>> > at >>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149) >>>> > at >>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181) >>>> > ... 8 more >>>> > Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856) >>>> ERROR: Syntax error at or near "," >>>> > ... 20 more >>>> > >>>> > 14/08/25 10:14:12 INFO mapreduce.Job: map 0% reduce 0% >>>> > 14/08/25 10:14:25 INFO mapreduce.Job: Task Id : >>>> attempt_1408941947869_0001_m_000000_1, Status : FAILED >>>> > Error: java.io.IOException: java.sql.SQLException: >>>> [Vertica][VJDBC](4856) ERROR: Syntax error at or near "," >>>> > at >>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184) >>>> > at >>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647) >>>> > at >>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770) >>>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) >>>> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) >>>> > at java.security.AccessController.doPrivileged(Native Method) >>>> > at javax.security.auth.Subject.doAs(Subject.java:415) >>>> > at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) >>>> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) >>>> > Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: >>>> Syntax error at or near "," >>>> > at com.vertica.util.ServerErrorData.buildException(Unknown >>>> Source) >>>> > at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown >>>> Source) >>>> > at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown >>>> Source) >>>> > at com.vertica.dataengine.VDataEngine.prepare(Unknown Source) >>>> > at com.vertica.dataengine.VDataEngine.prepare(Unknown Source) >>>> > at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source) >>>> > at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown >>>> Source) >>>> > at >>>> com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source) >>>> > at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source) >>>> > at >>>> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141) >>>> > at >>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149) >>>> > at >>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181) >>>> > ... 8 more >>>> > Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856) >>>> ERROR: Syntax error at or near "," >>>> > ... 20 more >>>> > >>>> > 14/08/25 10:14:34 INFO mapreduce.Job: Task Id : >>>> attempt_1408941947869_0001_m_000000_2, Status : FAILED >>>> > Error: java.io.IOException: java.sql.SQLSyntaxErrorException: >>>> [Vertica][VJDBC](4856) ERROR: Syntax error at or near "," >>>> > at >>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184) >>>> > at >>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647) >>>> > at >>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770) >>>> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) >>>> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) >>>> > at java.security.AccessController.doPrivileged(Native Method) >>>> > at javax.security.auth.Subject.doAs(Subject.java:415) >>>> > at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) >>>> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) >>>> > Caused by: java.sql.SQLSyntaxErrorException: [Vertica][VJDBC](4856) >>>> ERROR: Syntax error at or near "," >>>> > at com.vertica.util.ServerErrorData.buildException(Unknown >>>> Source) >>>> > at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown >>>> Source) >>>> > at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown >>>> Source) >>>> > at com.vertica.dataengine.VDataEngine.prepare(Unknown Source) >>>> > at com.vertica.dataengine.VDataEngine.prepare(Unknown Source) >>>> > at com.vertica.jdbc.common.SPreparedStatement.<init>(Unknown >>>> Source) >>>> > at com.vertica.jdbc.jdbc4.S4PreparedStatement.<init>(Unknown >>>> Source) >>>> > at >>>> com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl.<init>(Unknown Source) >>>> > at >>>> com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown Source) >>>> > at com.vertica.jdbc.common.SConnection.prepareStatement(Unknown >>>> Source) >>>> > at >>>> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141) >>>> > at >>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149) >>>> > at >>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181) >>>> > ... 8 more >>>> > Caused by: com.vertica.support.exceptions.SyntaxErrorException: >>>> [Vertica][VJDBC](4856) ERROR: Syntax error at or near "," >>>> > ... 21 more >>>> > >>>> > 14/08/25 10:14:43 INFO mapreduce.Job: map 100% reduce 0% >>>> > 14/08/25 10:14:44 INFO mapreduce.Job: Job job_1408941947869_0001 >>>> failed with state FAILED due to: Task failed >>>> task_1408941947869_0001_m_000000 >>>> > Job failed as tasks failed. failedMaps:1 failedReduces:0 >>>> > >>>> > 14/08/25 10:14:45 INFO mapreduce.Job: Counters: 9 >>>> > Job Counters >>>> > Failed map tasks=4 >>>> > Launched map tasks=4 >>>> > Other local map tasks=3 >>>> > Rack-local map tasks=1 >>>> > Total time spent by all maps in occupied slots >>>> (ms)=53177 >>>> > Total time spent by all reduces in occupied slots (ms)=0 >>>> > Total time spent by all map tasks (ms)=53177 >>>> > Total vcore-seconds taken by all map tasks=53177 >>>> > Total megabyte-seconds taken by all map tasks=54453248 >>>> > 14/08/25 10:14:45 WARN mapreduce.Counters: Group FileSystemCounters >>>> is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead >>>> > 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Transferred 0 bytes >>>> in 99.0548 seconds (0 bytes/sec) >>>> > 14/08/25 10:14:45 WARN mapreduce.Counters: Group >>>> org.apache.hadoop.mapred.Task$Counter is deprecated. Use >>>> org.apache.hadoop.mapreduce.TaskCounter instead >>>> > 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Exported 0 records. >>>> > 14/08/25 10:14:45 ERROR tool.ExportTool: Error during export: Export >>>> job failed! >>>> > Regards >>>> > Muthupandi.K >>>> > >>>> > >>>> > >>>> > <image002.png> >>>> > >>>> >>>> >>> >> >
