Hi Eva, After a close at the code, I think this is not a bug. We need to find out how to avoid this.
Thanks, Yongqiang On 09-9-9 下午1:31, "He Yongqiang" <heyongqi...@software.ict.ac.cn> wrote: > Hi Eva, > Can you open a new jira for this? And let’s discuss and resolve this > issue. > I guess this is because the partition metadata is added before the data is > available. > > Thanks > Yongqiang > On 09-9-9 下午1:18, "Eva Tse" <e...@netflix.com> wrote: > >> >> We are planning to start enabling ad-hoc querying on our hive warehouse and >> we tested some of the concurrent queries and found the following issue: >> >> Query 1 – doing ‘insert overwrite table yyy .... partition (dateint = xxx) >> select ... from yyy where dateint = xxx’ This is done to merge small files >> within a partition in table yyy >> Query 2 – doing some select on the same table joining another table. >> >> What we found is that query 2 would fail with the following exceptions in >> multiple reducers. >> java.io.FileNotFoundException: File does not exist: >> hdfs://ip-10-251-98-80.ec2.internal:9000/user/hive/dataeng/warehouse/nccp_ses >> sion_facts/dateint=20090908/hour=9/sessionsFacts_P20090909T021823L20090908T09 >> -r-00006 >> at >> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSys >> tem.java:457) >> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:671) >> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1417) >> at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1412) >> at >> org.apache.hadoop.mapred.SequenceFileRecordReader.(SequenceFileRecordReader.j >> ava:43) >> at >> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFile >> InputFormat.java:63) >> at >> org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat. >> java:236) >> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:336) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) >> at org.apache.hadoop.mapred.Child.main(Child.java:170) >> >> Is this expected? If so, is there a jira or is it planned to be addressed? We >> are trying to think of workaround, but haven’t thought of good ones as >> swapping of files would ideally be handled inside hive. >> >> Please let us know your feedback. >> >> Thanks, >> Eva. >
