[ https://issues.apache.org/jira/browse/AVRO-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069792#comment-13069792 ]
Doug Cutting commented on AVRO-806: ----------------------------------- Yes, CIF file looks promising. It's great to see all the benchmarks! I wonder if the advantages of CIF could be had without a custom HDFS block placement strategy? For example, one might pack the files of a split directory into a single file whose block size was set to the size of the file, forcing it into a single block. This would guarantee locality for the columns of a split. In other words, instead of groups of column-major records within a file ("block columnar" in Raymie's document) on one hand or a file-per-column on the other ("file columnar"), we have a single group per file. Since splits might often be bigger than RAM, creating these would probably require two steps: writing a set of temporary local files, one per column, then appending these into the final output. The file would have an index indicating where each column lies, and each column within the file would permit efficient skipping, in the style of CIF. > add a column-major codec for data files > --------------------------------------- > > Key: AVRO-806 > URL: https://issues.apache.org/jira/browse/AVRO-806 > Project: Avro > Issue Type: New Feature > Components: java, spec > Reporter: Doug Cutting > Assignee: Doug Cutting > Attachments: AVRO-806-v2.patch, AVRO-806.patch, avro-file-columnar.pdf > > > Define a codec that, when a data file's schema is a record schema, writes > blocks within the file in column-major order. This would permit better > compression and also permit efficient skipping of fields that are not of > interest. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira