[ https://issues.apache.org/jira/browse/SQOOP-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yulei Yang updated SQOOP-3262: ------------------------------ Attachment: sqoop_3262.patch > Duplicate rows found when split-by column is of type String > ----------------------------------------------------------- > > Key: SQOOP-3262 > URL: https://issues.apache.org/jira/browse/SQOOP-3262 > Project: Sqoop > Issue Type: Bug > Components: connectors/generic > Affects Versions: 1.4.6 > Reporter: Yulei Yang > Attachments: sqoop_3262.patch > > > When using string(or char) type column as split-by column, sometimes we found > duplicate rows, usually this is caused by source RMDBS is case insensitive > when do comparison. Here is a case, (split query sql): > 1. where id >='A' and id < 'E' > 2. where id >='a' and id < 'e' > if the RMDBS is CI, these two different split will get same result, thus > caused duplication. > By default oracle and db2 is CS, but a DBA can change it. so we need to check > it before import. > By default,sql server is CI,solution is --split-by '<your_column> collate > xxx_collation', like Chinese_PRC_Bin. > By default intersystems cachedb is CI,solution is --split-by > "%sqlstring(<your_column>)". > Mysql is CI by default, but “--split-by 'binary <your_column>' ” is throwing > below exception: > ERROR tool.ImportTool: Encountered IOException running import job: > java.io.IOException: Sqoop does not have the splitter for the given SQL data > type. Please use either different split column (argument --split-by) or lower > the number of mappers to 1. Unknown SQL data type: -3 > at > org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:165) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) > at > org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:196) > at > org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:169) > at > org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:266) > at > org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:729) > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:499) > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) > at org.apache.sqoop.Sqoop.run(Sqoop.java:143) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) > at org.apache.sqoop.Sqoop.main(Sqoop.java:236) > I have apply a patch for mysql's case. -- This message was sent by Atlassian JIRA (v6.4.14#64029)