Hi there
I had trouble with SQOOP export out of the box and have researched (ie.
googled) this one to death.
I wanted to publish the answer here for future generations as I myself couldn’t
find a solution and worked literally for hours with a colleague to investigate
these errors.
I configured SQOOP locally successfully, which I confirmed by calling ‘sqoop
version’:
[hadoop@ip-123-45-67-890 ~]$ sqoop version
15/06/19 09:07:29 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Sqoop 1.4.6
git commit id c0c5a81723759fa575844a0a1eae8f510fa32c25
Compiled by root on Mon Apr 27 14:38:36 CST 2015
Linux version: Amazon Linux AMI release 2015.03
Syntax:
sqoop export --connect "jdbc:sqlserver://***URL GOES
HERE***.us-west-2.rds.amazonaws.com:1433;databaseName=<databasename>" --table
<tablename> --export-dir
hdfs://<IPgoeshere>:<port>/user/hive/warehouse/<hive_file> --username
<username> --password <password>
15/06/19 08:44:33 INFO mapreduce.Job: Task Id :
attempt_1434676168391_0012_m_000001_0, Status : FAILED
Error: java.lang.ClassCastException: org.apache.hadoop.io.BytesWritable cannot
be cast to org.apache.hadoop.io.LongWritable
at
org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:95)
at
org.apache.sqoop.mapreduce.CombineShimRecordReader.getCurrentKey(CombineShimRecordReader.java:38)
at
org.apache.sqoop.mapreduce.CombineFileRecordReader.getCurrentKey(CombineFileRecordReader.java:79)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.getCurrentKey(MapTask.java:516)
at
org.apache.hadoop.mapreduce.task.MapContextImpl.getCurrentKey(MapContextImpl.java:70)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.getCurrentKey(WrappedMapper.java:81)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:152)
at
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170)
Kind Regards
Timothy Garza
[Collinson Group]<http://www.collinsongroup.com/>
The Collinson Group Limited; Registered number: 2577557, Registered in England
& Wales; Registered Office: Cutlers Exchange, 123 Houndsditch, London, EC3A 7BU.
This e-mail may contain privileged and confidential information and/or
copyright material and is intended for the use of the addressee only. If you
receive this e-mail by mistake please advise the sender immediately by using
the reply facility in your e-mail software and delete this e-mail from your
computer system. You may not deliver, copy or disclose its contents to anyone
else. Any unauthorised use may be unlawful. Any views expressed in this e-mail
are those of the individual sender and may not necessarily reflect the views of
The Collinson Group Ltd and/or its subsidiaries or any other associated company
(collectively “Collinson Group”).
As communications via the Internet are not secure Collinson Group cannot accept
any liability if this e-mail is accessed by third parties during the course of
transmission or is modified or amended in any way following despatch. Collinson
Group cannot guarantee that any attachment to this email does not contain a
virus, therefore it is strongly recommended that you carry out your own virus
check before opening any attachment, as we cannot accept liability for any
damage sustained as a result of software virus infection. Senders of messages
shall be taken to consent to the monitoring and recording of e-mails addressed
to members of the Company.
