The Avro file format supports appends. However some filesystems (e.g., HDFS and S3) may not support append, so applications generally avoid depending on it. Also, it can complicate application semantics when the contents of files change after they are first created.
The Java API supports append: https://avro.apache.org/docs/1.8.2/api/java/org/apache/avro/file/DataFileWriter.html#appendTo(java.io.File) There is also an open feature request to extend the concat tool to support append: https://issues.apache.org/jira/browse/AVRO-1856 Doug On Fri, Aug 25, 2017 at 12:10 PM, Matthew Stowe <matthew.st...@schneider-electric.com> wrote: > Hello Avro User Community, > > > > I am reviewing the specification for Object Container Files > (http://avro.apache.org/docs/1.8.2/spec.html#Object+Container+Files) and the > available Avro libraries from Apache and Microsoft. > > > > I am hoping the community can help clarify a question I have with respect to > Object Container files. > > > > My question is… > > Is it compliant with the specification to append objects to an Object > Container File after the header and some number of initial objects have been > written and the file closed? > > > > The reason I ask this question is that while the specification does not > explicitly state that appending to an Object Container File is not allowed, > none of the libraries I have evaluated support doing so. The libraries I > have looked at support creating a new Object Container File, writing the > header, some number of objects, and then closing it… but they do not support > coming back to that file at a later time and appending more objects. > > > > Is this simply a gap in the libraries or am I missing something in the > specification that states appending to Object Container Files is prohibited? > Or is this rather against some best practice that I am not aware of, such as > in HDFS? > > > > Cheers, > > Matt
