So we should ignore both . and _ ? Isn't . enough? ben
On Wednesday 23 January 2008 20:32:34 eric baldeschwieler (JIRA) wrote: > [ > https://issues.apache.org/jira/browse/PIG-71?page=com.atlassian.jira.plugin >.system.issuetabpanels:comment-tabpanel&focusedCommentId=12561934#action_125 >61934 ] > > eric baldeschwieler commented on PIG-71: > ---------------------------------------- > > There are two different issues confused here. > > 1) metadata files in output directories. Hadoop reserves the right to put > metadata in the output directory in files and directories that start with > '_'. Pig should behave consistently with this. We should document this in > Hadoop and Pig > > 2) speculative execution seems to be leaving task directories about. If > this is happening on successful runs, something is broken and should be > fixed. > > > Support for Hadoop Speculative Execution > > ---------------------------------------- > > > > Key: PIG-71 > > URL: https://issues.apache.org/jira/browse/PIG-71 > > Project: Pig > > Issue Type: New Feature > > Environment: Hadoop > > Reporter: Amir Youssefi > > Priority: Minor > > > > If Speculative Execution is used in Hadoop while creating a data-set then > > Pig scripts loading this data-set may fail. Reason is temp directories > > generated in the process. Pig can filter out these temp directories and > > problem gets solved. Here is sample error: [main] ERROR org.apache.pig - > > Error message from task (map) tip_..._0001_m_002735 java.io.EOFException > > at java.io.DataInputStream.readFully(DataInputStream.java:180) at > > org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java: > >56) at > > org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90) at > > org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:152 > >4) at > > org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile. > >java:1590) at > > org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.jav > >a:1626) at > > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712) at > > org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecord > >Reader.java:79) ...
