Thanks Venkat, I gave it a try with HCatalog today. Since --as-avrodatafile
is depricated, I'm not able generate Avro object and give it to Parquet
Avro Output format. So the job not able to initialized.
This is my Hive table
create external table hcat_test (id BIGINT, name BIGINT)
ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe'
STORED AS
INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat"
OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat"
LOCATION '/tmp/data/hcat_test';
my sqoop command
sqoop import --table test_db --username username --connect
"jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password
--driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name
--hcatalog-table hcat_employee
and I'm getting
14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area
hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003
Exception in thread "main" java.lang.InstantiationError:
org.apache.hadoop.mapreduce.JobContext
at
parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68)
at
org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135)
at
org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
at
org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
at
org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
I suspect the absence of Avro Object causing the issue. Any suggestions pls.
Regards,
Ananth.P,
On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan <
[email protected]> wrote:
> Sqoop already is integrated with HCatalog so many different storage types
> should be possible to be used with Sqoop as long as there is a hive serde
> for it. Check out the sqoop user guide on HCatalog integration.
>
>
> On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <[email protected]> wrote:
>
>> How can I extend Sqoop client to support custom output format or provide
>> Avro-Parquet support? Right now Sqoop support only Text and Sequence file
>> output format.
>>
>> Regards,
>> Ananth.P,
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <[email protected]>wrote:
>>
>>> How can I extend Sqoop client to support custom output format or provide
>>> Avro-Parquet support? Right now Sqoop support only Text and Sequence file
>>> output format.
>>>
>>> Regards,
>>> Ananth.P,
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <[email protected]>wrote:
>>>
>>>> How can I extend Sqoop client to support custom output format or
>>>> provide Avro-Parquet support? Right now Sqoop support only Text and
>>>> Sequence file output format.
>>>>
>>>>
>>>>
>>>> Regards,
>>>> Ananth.P,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
