[ https://issues.apache.org/jira/browse/AVRO-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187338#comment-13187338 ]
Scott Carey edited comment on AVRO-991 at 1/17/12 12:34 AM: ------------------------------------------------------------ To help out Tom's checksum issue, we could support user-provided sync markers. However, two files with the same logical content can differ in other ways too: User provided metadata, different block sizes, compression codecs, etc. That brings up another tool -- we could provide a tool that does checksums on the binary content of an avro data file. and ignores the sync markers. was (Author: scott_carey): TO help out Tom's md5 issue, we could support user-provided sync markers. However, two files with the same logical content can differ in other ways too: User provided metadata, different block sizes, compression codecs, etc. That brings up another tool -- we could provide a tool that does checksums on the binary content of an avro data file. and ignores the sync markers. > Allow combining multiple Avro files within a stream. (no files on disk) > ----------------------------------------------------------------------- > > Key: AVRO-991 > URL: https://issues.apache.org/jira/browse/AVRO-991 > Project: Avro > Issue Type: Improvement > Components: java > Affects Versions: 1.6.1 > Reporter: Frank Grimes > > It would be nice to be able to do as follows: > cat file1.avro file2.avro | java -jar avro-tools.jar streamcombine > > combined-file.avro > or similarly > > hadoop dfs -cat hdfs://hadoop/file1.avro hdfs://hadoop/file2.avro | java > -jar avro-tools.jar streamcombine | hdfs -put - > hdfs://hadoop/combined-file.avro > See the following thread for details: > http://mail-archives.apache.org/mod_mbox/avro-user/201201.mbox/%3cc08f1de9-97a8-4d28-b0ad-5e4a7f32f...@gmail.com%3e -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira