Hey man, Try adding --verbose. Normally Sqoop will let you know the records that are failing.
-Abe On Tue, Jul 28, 2015 at 6:25 PM, Eric <[email protected]> wrote: > Hi here, > > I'm trying to export some data from amazon S3 to postgresql. The data are > coming from redshift and the unload command. > > It works well with 20millions entries. However , with 800millions, I got > some errors right after the map-reduce job starts. I also tried with > different delimiters. The data are escaped with the backslash character, > and the fields are enclosed by double-quotes. > > My command is: > sqoop export --connect jdbc:postgresql://uri/db --username [] --table [] > --export-dir s3://[] --password '7dzeqwYb?WaqKMGPz8NA(y' > --fields-terminated-by ',' --enclosed-by '\"' --null-string "" > --null-non-string "" --escaped-by \\ > > I was wondering if there was a way to fine the entries causing this? I > tried to find in all the logs and couldn't find anything. > Do I miss anything? Is there a proper way to solve this? Thank you ! > > > The output : > Error: java.io.IOException: Can't export data, please check failed map > task logs > at > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112) > > at > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39) > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:152) > > at > org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170) > > Caused by: com.cloudera.sqoop.lib.RecordParser$ParseError: Expected > delimiter at position 235 > > at org.apache.sqoop.lib.RecordParser.parseRecord(RecordParser.java:319) > > at org.apache.sqoop.lib.RecordParser.parseRecord(RecordParser.java:108) > > at org.apache.sqoop.lib.RecordParser.parseRecord(RecordParser.java:125) > > at updates_backfill.parse(updates_backfill.java:1498) > > at > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83) > >
