This is the first time that I am using since I am having several problems with files with different characters. I was wondering that with Avro I could solve that. Other option that I thought was to change the code/decode on base64.
Since there are tickets like this: https://issues.apache.org/jira/browse/SQOOP-1283 https://issues.apache.org/jira/browse/SQOOP-1282 I think this is supposed to be working. 2015-05-05 21:13 GMT-03:00 Gwen Shapira <[email protected]>: > Looks like Sqoop never figured out that the file is Avro (you can see it > calls TextExportMapper and not AvroExportMapper). > > Sqoop guesses that the file is Avro based on the first 3 bytes (should be > 'Obj'), which actually look correct in your case... so, while I can tell > what is the issue, I can't help solve it :( > > Does it work on other Avro files? Did it work on older versions? > > Gwen > > > On Tue, May 5, 2015 at 1:50 PM, Henrique Andrade <[email protected]> > wrote: > >> I am trying to export a file to a PostgreSQL that is in Avro format but I >> am getting the following errors: >> >> >> I am using the 1.4.6 RC2 version. Any help? >> >> 2015-05-05 17:47:09,793 DEBUG - Fetching password from job credentials >> store >> 2015-05-05 17:47:10,271 ERROR - >> 2015-05-05 17:47:10,271 ERROR - Exception raised during data export >> 2015-05-05 17:47:10,271 ERROR - >> 2015-05-05 17:47:10,271 ERROR - Exception: >> java.lang.RuntimeException: Can't parse input data: '"sqlType":"93"}' >> at vw_vacation.__loadFromFields(vw_vacation.java:3096) >> at vw_vacation.parse(vw_vacation.java:2709) >> at >> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83) >> at >> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39) >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) >> at >> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) >> at >> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.lang.NumberFormatException: For input string: >> ""sqlType":"93"}" >> at >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> at java.lang.Integer.parseInt(Integer.java:481) >> at java.lang.Integer.valueOf(Integer.java:582) >> at vw_vacation.__loadFromFields(vw_vacation.java:2798) >> ... 13 more >> 2015-05-05 17:47:10,272 ERROR - On input: Obj >> avro.schema�.{"type":"record","name":"sqoop_import_VW_VENDAS_ITEMS_DETALHE","doc":"Sqoop >> import of >> VW_VENDAS_ITEMS_DETALHE","fields":[{"name":"CreatedAt","type":["null","long"],"default":null,"columnName":"CreatedAt","sqlType":"93"},{"name":"StoreId","type":["null","int"],"default":null,"columnName":"StoreId","sqlType":"5"},{"name":"ItemId","type":["null","long"],"default":null,"columnName":"ItemId","sqlType":"-5"},{"name":"StoreName","type":["null","string"],"default":null,"columnName":"StoreName","sqlType":"12"},{"name":"StoreCode","type":["null","string"],"default":null,"columnName":"StoreCode","sqlType":"12"},{"name":"OrderId","type":["null","string"],"default":null,"columnName":"OrderId","sqlType":"12"},{"name":"Status","type":["null","string"],"default":null,"columnName":"Status","sqlType":"12"},{"name":"CustomerId","type":["null","long"],"default":null,"columnName":"CustomerId","sqlType":"-5"},{"name":"CustomerFirstName","type":["null","string"],"default":null,"columnName":"CustomerFirstName","sqlType":"12"},{"name":"CustomerLastName","type":["null","string"],"default":null,"columnName":"CustomerLastName","sqlType":"12"},{"name":"CustomerEmail","type":["null","string"],"default":null,"columnName":"CustomerEmail","sqlType":"12"},{"name":"CustomerAddressType","type":["null","string"],"default":null,"columnName":"CustomerAddressType","sqlType":"12"},{"name":"CustomerStreet","type":["null","string"],"default":null,"columnName":"CustomerStreet","sqlType":"12"},{"name":"CustomerCity","type":["null","string"],"default":null,"columnName":"CustomerCity","sqlType":"12"},{"name":"CustomerZipCode","type":["null","string"],"default":null,"columnName":"CustomerZipCode","sqlType":"12"},{"name":"CustomerRegion","type":["null","string"],"default":null,"columnName":"CustomerRegion","sqlType":"12"},{"name":"CustomerCountry","type":["null","string"],"default":null,"columnName":"CustomerCountry","sqlType":"12"},{"name":"CustomerPhone","type":["null","string"],"default":null,"columnName":"CustomerPhone","sqlType":"12"},{"name":"ProductId","type":["null","long"],"default":null,"columnName":"ProductId","sqlType":"-5"},{"name":"SkuId","type":["null","string"],"default":null,"columnName":"SkuId","sqlType":"12"},{"name":"ProductName","type":["null","string"],"default":null,"columnName":"ProductName","sqlType":"12"},{"name":"CurrencyCode","type":["null","string"],"default":null,"columnName":"CurrencyCode","sqlType":"12"},{"name":"QtdInvoice","type":["null","string"],"default":null,"columnName":"QtdInvoice","sqlType":"3"},{"name":"QtdOrdered","type":["null","string"],"default":null,"columnName":"QtdOrdered","sqlType":"3"},{"name":"QtdReturned","type":["null","string"],"default":null,"columnName":"QtdReturned","sqlType":"3"},{"name":"OriginalPrice","type":["null","string"],"default":null,"columnName":"OriginalPrice","sqlType":"3"},{"name":"InvoicedTotalRow","type":["null","string"],"default":null,"columnName":"InvoicedTotalRow","sqlType":"3"}],"tableName":"VW_VENDAS_ITEMS_DETALHE"}��aV5jc >> � ۨ cu)p�� ����P \ BBeach Park >> 2015-05-05 17:47:10,272 ERROR - On input file: >> file:/Users/henriqueandrade/Documents/Workspace/sandbox/Spring Data/Chapter >> 06/Chapter-06/sqoop-test/src/main/resources/data/mysql/part-m-00000.avro >> 2015-05-05 17:47:10,272 ERROR - At position 0 >> 2015-05-05 17:47:10,272 ERROR - >> 2015-05-05 17:47:10,272 ERROR - Currently processing split: >> 2015-05-05 17:47:10,272 ERROR - >> Paths:/Users/henriqueandrade/Documents/Workspace/sandbox/Spring >> Data/Chapter >> 06/Chapter-06/sqoop-test/src/main/resources/data/mysql/part-m-00000.avro:0+217032,/Users/henriqueandrade/Documents/Workspace/sandbox/Spring >> Data/Chapter >> 06/Chapter-06/sqoop-test/src/main/resources/data/mysql/part-m-00001.avro:0+217032 >> 2015-05-05 17:47:10,272 ERROR - >> 2015-05-05 17:47:10,273 ERROR - This issue might not necessarily be >> caused by current input >> 2015-05-05 17:47:10,273 ERROR - due to the batching nature of export. >> 2015-05-05 17:47:10,273 ERROR - >> 2015-05-05 17:47:10,273 DEBUG - Instructing auto-progress thread to quit. >> 2015-05-05 17:47:10,273 DEBUG - Waiting for progress thread shutdown... >> 2015-05-05 17:47:10,273 INFO - Auto-progress thread is finished. >> keepGoing=false >> 2015-05-05 17:47:10,273 DEBUG - Progress thread shutdown detected. >> 2015-05-05 17:47:10,274 DEBUG - Committing transaction of 0 statements >> 2015-05-05 17:47:10,275 INFO - map task executor complete. >> 2015-05-05 17:47:10,275 WARN - job_local1317842886_0001 >> java.lang.Exception: java.io.IOException: Can't export data, please check >> failed map task logs >> at >> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) >> at >> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) >> Caused by: java.io.IOException: Can't export data, please check failed >> map task logs >> at >> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112) >> at >> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39) >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) >> at >> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) >> at >> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.lang.RuntimeException: Can't parse input data: >> '"sqlType":"93"}' >> at vw_vacation.__loadFromFields(vw_vacation.java:3096) >> at vw_vacation.parse(vw_vacation.java:2709) >> at >> org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83) >> ... 11 more >> Caused by: java.lang.NumberFormatException: For input string: >> ""sqlType":"93"}" >> at >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> at java.lang.Integer.parseInt(Integer.java:481) >> at java.lang.Integer.valueOf(Integer.java:582) >> at vw_vacation.__loadFromFields(vw_vacation.java:2798) >> ... 13 more >> 2015-05-05 17:47:10,280 DEBUG - PrivilegedAction as:henriqueandrade >> (auth:SIMPLE) >> from:org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:331) >> 2015-05-05 17:47:10,679 DEBUG - PrivilegedAction as:henriqueandrade >> (auth:SIMPLE) >> from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322) >> 2015-05-05 17:47:10,680 INFO - Job job_local1317842886_0001 running in >> uber mode : false >> 2015-05-05 17:47:10,681 INFO - map 0% reduce 0% >> > >
