[ 
https://issues.apache.org/jira/browse/AVRO-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760356#action_12760356
 ] 

Doug Cutting commented on AVRO-127:
-----------------------------------

In Avro 1.1 we added DataFileWriter#addSchema(Schema):

http://hadoop.apache.org/avro/docs/current/api/java/org/apache/avro/file/DataFileWriter.html#addSchema%28org.apache.avro.Schema%29

This only works if the top-level schema is a union.  It lets you add clauses to 
that union as you write the file.  So, for example, you might do something like:

{noformat}
DataFileWriter<Object> out = 
  new DataFileWriter<Object>(Schema.createUnion(new ArrayList()[])),
   new FileOutputStream("foo"), new GenericDatumWriter());
...
out.addSchema(x.getSchema());
out.write(x);
...
out.addSchema(y.getSchema());
out.write(y)
...
{noformat}

Does that achieve what you want?  This should work with specific or reflect 
data too.

Note that you should only add a given schema once.  Also, if you have a large 
number of schemas (>10?) per file then performance may lag a bit, since it 
scans the union linearly each time an entry is appended.  But that could easily 
be optimized if it became a problem.

I am tempted to make this the default for DataFileWriter: that we add a 
constructor that does not include a schema, in which case an empty union is 
used.  Also, we could then automatically add schemas to the union if they're 
not already present.  Would that be useful?

>  Avro should support multiple schemas from the same AVRO file 
> --------------------------------------------------------------
>
>                 Key: AVRO-127
>                 URL: https://issues.apache.org/jira/browse/AVRO-127
>             Project: Avro
>          Issue Type: New Feature
>         Environment: all systems
>            Reporter: Yang Yang
>
> in our application, we often have to merge together all the day's data into 
> one daily file,
> data file schemas must be changing within the day, so we have different 
> schemas within the same file.
> AVRO should support multiple schemas within the same file, ideally different 
> schemas can be parsed out
> to different children classes of the same schema class.
>               

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to