[ https://issues.apache.org/jira/browse/AVRO-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13603413#comment-13603413 ]
Yin Huai commented on AVRO-1208: -------------------------------- Here are some comparisons between RCFile and Trevni. I mainly focus on the data reading. Based on current implementations, RCFile and Trevni have three major differences. # The row group size in RCFile is configurable. Trevni uses a single row group for a Trevni file (so applications need to horizontally partition the table to multiple Trevni files, and the size of a Trevni file needs to be less than the HDFS block size. Am i right?). # When reading needed columns of a row group, RCFile loads these columns at once. So, applications need to wait the I/O (on reading all needed columns in a row group) before accessing any row in this row group. In a Trevni file, a column is stored by many small blocks which are compression units. When Trevni needs to read data from disks, applications only wait for Trevni to read a few blocks before accessing a row. # When reading needed columns of a row group, RCFile loads these columns in a column by column fashion. For Trevni, applications need to decide how to read needed columns. They can read data in a row by row fashion or in a column by column fashion. For a given table, applications need to set a suitable row group size for RCFile. A small row group size will cause a small size of a column in a row group (a column in a row group is stored contiguously). Many seeks will degrade the performance of data reading (this is described in Trevni specification). Also, a small row group size can cause a read buffer contain data from unneeded columns and cause the OS readahead less effective (cannot asynchronous fetch data from needed columns). To overcome the low I/O efficiency of RCFile, a large row group size can be used. However, RCFile needs to read all needed columns of a row group at once. In this way, CPU and I/O may not be effectively overlapped (less benefit from OS asynchronous readahead). Suppose that applications explicitly stores a table to multiple RCFile files and every file has a single row group. When applications process data in a file, it will be blocked until all needed data is loaded from disks. In this example, we will first wait on I/O and then wait on CPU. For the third difference, a large row group size in RCFile imply a higher I/O performance since all needed columns in a row group are read in a column by column fashion. But for Trevni, since applications usually read data in needed columns in a row by row fashion (seems AvroColumnReader reads data in a row by row fashion, and Hive and Pig integration of Trevni relies on this reader), the throughput of reading data stored in Trevni can be significantly degraded (cased by unnecessary disk seeks). > Improve Trevni's performance on row-oriented data access > -------------------------------------------------------- > > Key: AVRO-1208 > URL: https://issues.apache.org/jira/browse/AVRO-1208 > Project: Avro > Issue Type: Improvement > Affects Versions: 1.7.3 > Reporter: Yin Huai > Assignee: Yin Huai > Attachments: AVRO-1208.1.patch, AVRO-1208.2.patch > > > Trevni uses an 64KB internal buffer to store values of a column. When > accessing a column, it reads 64KB (if we do not consider compression and > checksum) data from the storage layer. However, when the table is accessed in > a row-oriented fashion (a entire row needs to be handed over to the upper > layer), in the worst case (a full table scan and values of this table are all > the same size), every 64KB data read can cause a seek. > This jira is used to discuss if we should consider the data access pattern > mentioned above and if so, how to improve the performance of Trevni. > Row-oriented data processing engines, e.g. Hive, can benefit from this work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira