To my best knowledge the argument --as-avrodatafile is not deprecated - why do you think so Ananth?
Jarcec On Fri, Apr 25, 2014 at 03:43:01PM -0700, Ananth Durai wrote: > Thanks Venkat, I gave it a try with HCatalog today. Since --as-avrodatafile > is depricated, I'm not able generate Avro object and give it to Parquet > Avro Output format. So the job not able to initialized. > > This is my Hive table > > create external table hcat_test (id BIGINT, name BIGINT) > ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe' > STORED AS > INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat" > OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat" > LOCATION '/tmp/data/hcat_test'; > > my sqoop command > > sqoop import --table test_db --username username --connect > "jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password > --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name > --hcatalog-table hcat_employee > > > and I'm getting > > 14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area > hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003 > Exception in thread "main" java.lang.InstantiationError: > org.apache.hadoop.mapreduce.JobContext > at > parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68) > at > org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135) > at > org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984) > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) > at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:566) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596) > at > org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186) > at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159) > at > org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239) > at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600) > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413) > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502) > at org.apache.sqoop.Sqoop.run(Sqoop.java:145) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) > at org.apache.sqoop.Sqoop.main(Sqoop.java:238) > > > > > I suspect the absence of Avro Object causing the issue. Any suggestions pls. > > > > > > > > Regards, > Ananth.P, > > > > > > > > On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan < > [email protected]> wrote: > > > Sqoop already is integrated with HCatalog so many different storage types > > should be possible to be used with Sqoop as long as there is a hive serde > > for it. Check out the sqoop user guide on HCatalog integration. > > > > > > On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <[email protected]> wrote: > > > >> How can I extend Sqoop client to support custom output format or provide > >> Avro-Parquet support? Right now Sqoop support only Text and Sequence file > >> output format. > >> > >> Regards, > >> Ananth.P, > >> > >> > >> > >> > >> > >> > >> > >> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <[email protected]>wrote: > >> > >>> How can I extend Sqoop client to support custom output format or provide > >>> Avro-Parquet support? Right now Sqoop support only Text and Sequence file > >>> output format. > >>> > >>> Regards, > >>> Ananth.P, > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <[email protected]>wrote: > >>> > >>>> How can I extend Sqoop client to support custom output format or > >>>> provide Avro-Parquet support? Right now Sqoop support only Text and > >>>> Sequence file output format. > >>>> > >>>> > >>>> > >>>> Regards, > >>>> Ananth.P, > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>> > >> > > > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > > to which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender immediately > > and delete it from your system. Thank You.
signature.asc
Description: Digital signature
