will it be added in to Sqoop road map anytime soon? Regards, Ananth.P,
On Mon, May 5, 2014 at 12:41 PM, Venkat Ranganathan < [email protected]> wrote: > yes, you can't use --as-avrodatafile with --hcatalog-table option. (In > fact you can't use --hive-import also with hcatalog options). I have > not myself tried paraquet format with hcatalog-table option though > > Thanks > > Venkat > > > On Mon, May 5, 2014 at 11:33 AM, Ananth Durai <[email protected]> wrote: > >> Jarek, Surprisingly --as-avrodatafile is not supported for HCat export. >> When I tried I got error like, >> >> HCatalog job is not compatible with AVRO format option as-avrodatafile >> option. >> >> Regards, >> Ananth.P, >> >> >> >> >> >> >> >> On Sun, May 4, 2014 at 1:55 PM, Jarek Jarcec Cecho <[email protected]>wrote: >> >>> To my best knowledge the argument --as-avrodatafile is not deprecated - >>> why do you think so Ananth? >>> >>> Jarcec >>> >>> On Fri, Apr 25, 2014 at 03:43:01PM -0700, Ananth Durai wrote: >>> > Thanks Venkat, I gave it a try with HCatalog today. Since >>> --as-avrodatafile >>> > is depricated, I'm not able generate Avro object and give it to Parquet >>> > Avro Output format. So the job not able to initialized. >>> > >>> > This is my Hive table >>> > >>> > create external table hcat_test (id BIGINT, name BIGINT) >>> > ROW FORMAT SERDE 'parquet.hive.serde.ParquetHiveSerDe' >>> > STORED AS >>> > INPUTFORMAT "parquet.hive.DeprecatedParquetInputFormat" >>> > OUTPUTFORMAT "parquet.hive.DeprecatedParquetOutputFormat" >>> > LOCATION '/tmp/data/hcat_test'; >>> > >>> > my sqoop command >>> > >>> > sqoop import --table test_db --username username --connect >>> > "jdbc:sqlserver://<server>:1433;DatabaseName=db" --password password >>> > --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --columns id,name >>> > --hcatalog-table hcat_employee >>> > >>> > >>> > and I'm getting >>> > >>> > 14/04/25 15:15:40 INFO mapred.JobClient: Cleaning up the staging area >>> > >>> hdfs://localhost.localdomain:8020/user/cloudera/.staging/job_201404250915_0003 >>> > Exception in thread "main" java.lang.InstantiationError: >>> > org.apache.hadoop.mapreduce.JobContext >>> > at >>> > >>> parquet.hive.DeprecatedParquetOutputFormat.checkOutputSpecs(DeprecatedParquetOutputFormat.java:68) >>> > at >>> > >>> org.apache.hcatalog.mapreduce.FileOutputFormatContainer.checkOutputSpecs(FileOutputFormatContainer.java:135) >>> > at >>> > >>> org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72) >>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984) >>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945) >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > at javax.security.auth.Subject.doAs(Subject.java:396) >>> > at >>> > >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >>> > at >>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945) >>> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:566) >>> > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596) >>> > at >>> > >>> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186) >>> > at >>> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159) >>> > at >>> > >>> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239) >>> > at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600) >>> > at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413) >>> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502) >>> > at org.apache.sqoop.Sqoop.run(Sqoop.java:145) >>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) >>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) >>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) >>> > at org.apache.sqoop.Sqoop.main(Sqoop.java:238) >>> > >>> > >>> > >>> > >>> > I suspect the absence of Avro Object causing the issue. Any >>> suggestions pls. >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > Regards, >>> > Ananth.P, >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > On Thu, Apr 24, 2014 at 6:25 PM, Venkat Ranganathan < >>> > [email protected]> wrote: >>> > >>> > > Sqoop already is integrated with HCatalog so many different storage >>> types >>> > > should be possible to be used with Sqoop as long as there is a hive >>> serde >>> > > for it. Check out the sqoop user guide on HCatalog integration. >>> > > >>> > > >>> > > On Thu, Apr 24, 2014 at 3:34 PM, Ananth Durai <[email protected]> >>> wrote: >>> > > >>> > >> How can I extend Sqoop client to support custom output format or >>> provide >>> > >> Avro-Parquet support? Right now Sqoop support only Text and >>> Sequence file >>> > >> output format. >>> > >> >>> > >> Regards, >>> > >> Ananth.P, >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> On Thu, Apr 24, 2014 at 3:31 PM, Ananth Durai <[email protected] >>> >wrote: >>> > >> >>> > >>> How can I extend Sqoop client to support custom output format or >>> provide >>> > >>> Avro-Parquet support? Right now Sqoop support only Text and >>> Sequence file >>> > >>> output format. >>> > >>> >>> > >>> Regards, >>> > >>> Ananth.P, >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> On Thu, Apr 24, 2014 at 3:26 PM, Ananth Durai <[email protected] >>> >wrote: >>> > >>> >>> > >>>> How can I extend Sqoop client to support custom output format or >>> > >>>> provide Avro-Parquet support? Right now Sqoop support only Text >>> and >>> > >>>> Sequence file output format. >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> Regards, >>> > >>>> Ananth.P, >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>> >>> > >> >>> > > >>> > > CONFIDENTIALITY NOTICE >>> > > NOTICE: This message is intended for the use of the individual or >>> entity >>> > > to which it is addressed and may contain information that is >>> confidential, >>> > > privileged and exempt from disclosure under applicable law. If the >>> reader >>> > > of this message is not the intended recipient, you are hereby >>> notified that >>> > > any printing, copying, dissemination, distribution, disclosure or >>> > > forwarding of this communication is strictly prohibited. If you have >>> > > received this communication in error, please contact the sender >>> immediately >>> > > and delete it from your system. Thank You. >>> >> >> > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
