[ 
https://issues.apache.org/jira/browse/AVRO-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760430#action_12760430
 ] 

Doug Cutting commented on AVRO-127:
-----------------------------------

This technique relies on the way that unions are encoded.  In general, if new 
elements are added to the end of a union schema then data written with this 
schema before these were added can still be read.  So you can do the same thing 
with GenericDatumWriter: just create the new extended union schema, then either 
use DatumWriter#setSchema() to install it or create a new DatumWriter with the 
new extended schema.

Note that you should store the union schema somewhere with the data, so that 
you can be guaranteed to be able to read it later.  Avro's data file format 
takes care of this for you.

> I normally use GenericDatumWriter so that I can use it directly on HDFS 
> ,instead of facing a file 
> (the hadoop FileInput/FileOutputFormat API gives me a DataInputStream to work 
> with, not a file)

Avro's DataFileWriter constructor accepts an OutputStream, and, with a bit of 
code you can implement Avro's SeekableInput interface so that you can read an 
Avro data file from HDFS:

{noformat}
public class SeekableHadoopInput extends FilterInputStream implements 
SeekableInput {
  public SeekableHadoopInput(FSDataInputStream in, long length) { 
    super(in);
    this.length = length;
  }
  public long length() { return length; }
  public void seek(long pos) { ((FSDataInputStream)in).seek(pos); }
  public long tell() { return ((FSDataInputStream)in).getPos(); }
}
{noformat}

Alternately, in Hadoop 0.21 you can store Avro data in SequenceFiles.  But we 
do really need to create InputFormat and OutputFormat's for Avro's data file...


>  Avro should support multiple schemas from the same AVRO file 
> --------------------------------------------------------------
>
>                 Key: AVRO-127
>                 URL: https://issues.apache.org/jira/browse/AVRO-127
>             Project: Avro
>          Issue Type: New Feature
>         Environment: all systems
>            Reporter: Yang Yang
>
> in our application, we often have to merge together all the day's data into 
> one daily file,
> data file schemas must be changing within the day, so we have different 
> schemas within the same file.
> AVRO should support multiple schemas within the same file, ideally different 
> schemas can be parsed out
> to different children classes of the same schema class.
>               

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to