[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-3015:
-------------------------------
Attachment: Test.tar.gz
I am uploading a test program that I wrote. What it does are:
* Reads an Avro file and prints out records to stdout.
* Each time when it prints out records, it also prints out the current sync
position (i.e. tell()).
* When encountering an exception during a next(), it does sync(tell() + 1).
Unfortunately, it doesn't seem that Avro files have sync positions between
records. Instead, it only has a single sync position at the end of file.
You can run my program as follows:
* Untar Test.tar.gz:
{code}
bad.avro
good.avro
Main.java
Test.jar
{code}
* There are two Avro files: good.avro and bad.avro:
{code:title=good.avro}
java -jar avro-tool.jar tojson good.avro
"0"
...
"999"
{code}
{code:title=bad.avro}
java -jar avro-tool.jar tojson bad.avro
Exception in thread "main" org.apache.avro.AvroRuntimeException:
java.io.IOException: Invalid sync!
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:210)
at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:64)
at org.apache.avro.tool.Main.run(Main.java:74)
at org.apache.avro.tool.Main.main(Main.java:63)
The Test.tar.gz contains the following files:
{code}
* Run "java -jar Test.jar good.avro" gives the following output:
{code}
lengh(): 3969
tell(): 3969
next(): 0
...
next(): 999
tell(): 3969
{code}
As can be seen, the sync position never changes between records.
* Run "java -jar Test.jar bad.avro" gives the following output:
{code}
lengh(): 3969
tell(): 3969
hasNext() or next() failed
tell(): 3970
{code}
The sync(tell()+1) sets the sync position to past the end of file, so it ends
the program.
In summary, I don't think that we can recover from a bad sync() in Avro. Please
correct me if my test program has a bug.
Thanks!
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
> Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch,
> PIG-3015-5.patch, Test.tar.gz
>
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni (as
> TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
