Hi
I am having sqoop import issue with OID (String) as splitby parameter.
*Sqoop Command : *
$SQOOP_HOME/bin/sqoop import --connect jdbc:oracle:thin:@HOST:POST/ORACLE_DB
--table IOC --null-string '\\N' --null-non-string '\\N' --username *****
--password ******* --hive-import --hive-table HIVE_DB.IOC
--fields-terminated-by "\\0001" --lines-terminated-by '\n' --split-by OID
--m 8 --verbose --delete-target-dir --target-dir /tmp/3/IOC
*Logs : *
.....
.....
.....
15/07/15 16:01:23 DEBUG mapreduce.JobBase: Adding to job classpath:
file:/usr/share/hadoop_echosystem/sqoop-1.4.5/lib/tdgssconfig.jar
15/07/15 16:01:26 DEBUG db.DBConfiguration: Fetching password from job
credentials store
15/07/15 16:01:27 INFO db.DBInputFormat: Using read commited transaction
isolation
15/07/15 16:01:27 INFO db.DataDrivenDBInputFormat: BoundingValsQuery:
SELECT MIN(OID), MAX(OID) FROM IOC
15/07/15 16:01:27 WARN db.TextSplitter: Generating splits for a textual
index column.
15/07/15 16:01:27 WARN db.TextSplitter: If your database sorts in a
case-insensitive order, this may result in a partial import or duplicate
records.
15/07/15 16:01:27 WARN db.TextSplitter: You are strongly encouraged to
choose an integral split column.
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= '00131DB9913442C3B7735411298DD302'' and upper
bound 'OID < '00131DB9''
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= '00131DB9'' and upper bound 'OID < '2쀲쀳ꀲ恂ꁁ''
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= '2쀲쀳ꀲ恂ꁁ'' and upper bound 'OID < '5耵耶䀲쀱쁁䁁쀻''
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= '5耵耶䀲쀱쁁䁁쀻'' and upper bound 'OID < '8䀸䀸ꀲ‿ꀼ''
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= '8䀸䀸ꀲ‿ꀼ'' and upper bound 'OID < ';;;耲耲耾聁耽''
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= ';;;耲耲耾聁耽'' and upper bound 'OID < '=쀽쀾′怲⁁怾''
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= '=쀽쀾′怲⁁怾'' and upper bound 'OID < '@聀聀쀲䀳䀻쁁䀿''
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= '@聀聀쀲䀳䀻쁁䀿'' and upper bound 'OID < 'C䁃䁃怲″ꀺ恁⁀''
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= 'C䁃䁃怲″ꀺ恁⁀'' and upper bound 'OID < 'FFF249AB''
15/07/15 16:01:27 DEBUG db.DataDrivenDBInputFormat: Creating input split
with lower bound 'OID >= 'FFF249AB'' and upper bound 'OID <=
'FFF249ABE3DE4841A88978978AEDC1B3''
Error: java.io.IOException: SQLException in nextKeyValue
at
org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
at
org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
.......
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.sql.SQLSyntaxErrorException: ORA-00911: invalid character
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
........
at
org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
... 12 more
Error: java.io.IOException: SQLException in nextKeyValue
at
org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
....
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.sql.SQLSyntaxErrorException: ORA-00911: invalid character
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:447)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:396)
.......
at
org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
... 12 more
Error: java.lang.RuntimeException: java.lang.RuntimeException:
java.sql.SQLRecoverableException: No more data to read from socket
at
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
.......
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.RuntimeException: java.sql.SQLRecoverableException: No
more data to read from socket
at
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at
org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: java.sql.SQLRecoverableException: No more data to read from
socket
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1157)
......
at
org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at
org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Error: java.io.IOException: SQLException in nextKeyValue
at
org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:553)
at
org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
......
at
org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)
... 12 more
15/07/15 16:08:07 INFO mapreduce.ImportJobBase: Transferred 7.6633 MB in
403.9478 seconds (19.4264 KB/sec)
15/07/15 16:08:07 INFO mapreduce.ImportJobBase: Retrieved 18165 records.
15/07/15 16:08:07 DEBUG util.ClassLoaderStack: Restoring classloader:
sun.misc.Launcher$AppClassLoader@4821e115
15/07/15 16:08:07 ERROR tool.ImportTool: Error during import: Import job
failed!
Through , It is working fine for other tables with String Column at SplitBy
parameter. But it is behaving strange for this sqoop import.
I see internal it uses "SELECT MIN(OID), MAX(OID) FROM IOC" for find min
and max values that will be used for destributing records among mappers.
It is strange to see '2쀲쀳ꀲ恂ꁁ'' , '5耵耶䀲쀱쁁䁁쀻' etc intemediate values for
OID column in logs. I don't know how its been derived with invalid
character which are actually causing issue.
I tried the same sql on oracle. It is giving values.
*MIN : 00131DB9913442C3B7735411298DD302**MAX :
FFF249ABE3DE4841A88978978AEDC1B3*
Help me in upderstanding the issue and resolving the same.
Regards
Sanjiv Singh
Mob : +091 9990-447-339
