[ https://issues.apache.org/jira/browse/HIVE-1332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133587#comment-13133587 ]
Steven Wong commented on HIVE-1332: ----------------------------------- If an archive operation runs when a select query is already running, the select may fail, right? > Archiving partitions > -------------------- > > Key: HIVE-1332 > URL: https://issues.apache.org/jira/browse/HIVE-1332 > Project: Hive > Issue Type: New Feature > Components: Metastore > Reporter: Paul Yang > Assignee: Paul Yang > Fix For: 0.6.0 > > Attachments: HIVE-1332.1.patch, HIVE-1332.2.patch, HIVE-1332.3.patch, > HIVE-1332.4.patch, HIVE-1332.5.patch, HIVE-1332.6.patch > > > Partitions and tables in Hive typically consist of many files on HDFS. An > issue is that as the number of files increase, there will be higher > memory/load requirements on the namenode. Partitions in bucketed tables are a > particular problem because they consist of many files, one for each of the > buckets. > One way to drastically reduce the number of files is to use hadoop archives: > http://hadoop.apache.org/common/docs/current/hadoop_archives.html > This feature would introduce an ALTER TABLE <table_name> ARCHIVE PARTITION > <spec> that would automatically put the files for the partition into a HAR > file. We would also have an UNARCHIVE option to convert the files in the > partition back to the original files. Archived partitions would be slower to > access, but they would have the same functionality and decrease the number of > files drastically. Typically, only seldom accessed partitions would be > archived. > Hadoop archives are still somewhat new, so we'll only put in support for the > latest released major version (0.20). Here are some bug fixes: > https://issues.apache.org/jira/browse/HADOOP-6591 (Important - could > potentially cause data loss without this fix) > https://issues.apache.org/jira/browse/HADOOP-6645 > https://issues.apache.org/jira/browse/MAPREDUCE-1585 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira