[ https://issues.apache.org/jira/browse/SQOOP-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134942#comment-16134942 ]
Zoltán Tóth commented on SQOOP-3139: ------------------------------------ New Patch was created after code review findings. > sqoop tries to re execute select query during import in case of a connection > reset error and this is causing lots of duplicate records from source > -------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: SQOOP-3139 > URL: https://issues.apache.org/jira/browse/SQOOP-3139 > Project: Sqoop > Issue Type: Bug > Affects Versions: 1.4.6 > Environment: IBM Hadoop distribution: 4.2.0 > version: 1.4.6_IBM_27 > Sqoop 1.4.6_IBM_27 > Reporter: hemanth meka > Assignee: Zoltán Tóth > Attachments: SQOOP-3139.patch, SQOOP-3139.patch, SQOOP-3139.patch > > > We cannot reproduce this issue as it depends on network. Here is a edited log > content for understanding the issue. > Log start > . > . > 2017-02-22 07:35:37,638 INFO [main] > org.apache.sqoop.mapreduce.sqlserver.SqlServerRecordReader: Using query: > select sequence_number, analytical_bundle_masked where ( Sequence_Number >= > 8571429 ) AND ( Sequence_Number <= 10000000 ) > 2017-02-22 07:35:37,662 INFO [main] > org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select > sequence_number, analytical_bundle_masked where ( Sequence_Number >= 8571429 > ) AND ( Sequence_Number <= 10000000 ) > 2017-02-22 07:39:00,533 ERROR [main] > org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: > com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset > at > com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399) > at > com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2383) > at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1884) > at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6685) > at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:6595) > at > com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:6571) > at com.microsoft.sqlserver.jdbc.TDSReader.readBytes(IOBuffer.java:6864) > at > com.microsoft.sqlserver.jdbc.TDSReader.readWrappedBytes(IOBuffer.java:6886) > at > com.microsoft.sqlserver.jdbc.TDSReader.readUnsignedShort(IOBuffer.java:6801) > at > com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValuePrep(dtv.java:3570) > at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:3936) > at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:226) > at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:144) > at > com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2099) > at > com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2084) > at > com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2427) > at > org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71) > at > com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61) > at QueryResult.readFields0(QueryResult.java:10706) > at QueryResult.readFields(QueryResult.java:10415) > at > org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244) > at > org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader.nextKeyValue(SQLServerDBRecordReader.java:148) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556) > at > org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80) > at > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at > org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 2017-02-22 07:39:00,552 WARN [main] > org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader: Trying to recover from > DB read failure: > java.io.IOException: SQLException in nextKeyValue > at > org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277) > at > org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader.nextKeyValue(SQLServerDBRecordReader.java:148) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556) > at > org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80) > at > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at > org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset > at > com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399) > at > com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2383) > at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1884) > at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6685) > at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:6595) > at > com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:6571) > at com.microsoft.sqlserver.jdbc.TDSReader.readBytes(IOBuffer.java:6864) > at > com.microsoft.sqlserver.jdbc.TDSReader.readWrappedBytes(IOBuffer.java:6886) > at > com.microsoft.sqlserver.jdbc.TDSReader.readUnsignedShort(IOBuffer.java:6801) > at > com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValuePrep(dtv.java:3570) > at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:3936) > at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:226) > at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:144) > at > com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2099) > at > com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2084) > at > com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2427) > at > org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71) > at > com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61) > at QueryResult.readFields0(QueryResult.java:10706) > at QueryResult.readFields(QueryResult.java:10415) > at > org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244) > ... 13 more > . > . > 2017-02-22 07:39:00,928 INFO [main] > org.apache.sqoop.mapreduce.sqlserver.SqlServerRecordReader: Using query: > select sequence_number, analytical_bundle_masked where ( Sequence_Number >= > 8571429 ) AND ( Sequence_Number <= 10000000 ) > 2017-02-22 07:39:00,929 INFO [main] > org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select > sequence_number, analytical_bundle_masked where ( Sequence_Number >= 8571429 > ) AND ( Sequence_Number <= 10000000 ) > 2017-02-22 08:05:03,199 INFO [Thread-15] > org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is > finished. keepGoing=false > 2017-02-22 08:05:03,295 INFO [main] org.apache.hadoop.mapred.Task: > Task:attempt_1486732304049_0615_m_000006_0 is done. And is in the process of > committing > 2017-02-22 08:05:03,344 INFO [main] org.apache.hadoop.mapred.Task: Task > attempt_1486732304049_0615_m_000006_0 is allowed to commit now > 2017-02-22 08:05:03,350 INFO [main] > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of > task 'attempt_1486732304049_0615_m_000006_0' to > hdfs://ehaascluster/jcrew/hdfs/qa/secure/work/customer/experian/experian_jb_analytical_bundle/_temporary/1/task_1486732304049_0615_m_000006 > 2017-02-22 08:05:03,390 INFO [main] org.apache.hadoop.mapred.Task: Task > 'attempt_1486732304049_0615_m_000006_1' done. > End of LogType:syslog > Container: container_e58_1486732304049_0615_01_000005 on > bi4c-11665-data-1.bi.services.bluemix.net_45454 > ========================================================================================================== > LogType:stderr > Log Upload Time:Wed Feb 22 08:07:25 +0000 2017 > LogLength:243 > Log Contents: > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.metrics2.impl.MetricsSystemImpl). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > End of LogType:stderr > LogType:stdout > Log Upload Time:Wed Feb 22 08:07:25 +0000 2017 > LogLength:0 > Log Contents: > End of LogType:stdout > LogType:syslog > Log Upload Time:Wed Feb 22 08:07:25 +0000 2017 > LogLength:104358 -- This message was sent by Atlassian JIRA (v6.4.14#64029)