[ https://issues.apache.org/jira/browse/HADOOP-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590692#comment-15590692 ]
VITALIY SAVCHENKO commented on HADOOP-13725: -------------------------------------------- {code} 1. MapFile.Writer w = null; try { w = new MapFile.Writer( new Configuration(), new Path("hdfs://192.168.56.101:9000/20161020/data"), MapFile.Writer.keyClass(IntWritable.class), MapFile.Writer.valueClass(IntWritable.class) ); w.append(new IntWritable(0), new IntWritable(100)); w.append(new IntWritable(10), new IntWritable(200)); w.append(new IntWritable(5), new IntWritable(400)); //java.io.IOException: key out of order: 5 after 10 } catch (Exception ex) { ex.printStackTrace(); } finally { w.close(); } MapFile.Reader reader = new MapFile.Reader( new Path("hdfs://192.168.56.101:9000/20161020/data"), new Configuration() ); System.out.println(reader.get(new IntWritable(0), new IntWritable())); //print 100 System.out.println(reader.get(new IntWritable(10), new IntWritable())); // print 200 MapFile corrent reader.close(); 2. Open MapFile for apped MapFile.Writer w = null; try { w = new MapFile.Writer( new Configuration(), new Path("hdfs://192.168.56.101:9000/20161020/data"), MapFile.Writer.keyClass(IntWritable.class), MapFile.Writer.valueClass(IntWritable.class) ); w.append(new IntWritable(0), new IntWritable(100)); w.append(new IntWritable(10), new IntWritable(200)); w.close(); w = new MapFile.Writer( new Configuration(), new Path("hdfs://192.168.56.101:9000/20161020/data"), SequenceFile.Writer.appendIfExists(true), // append to exist MapFile SequenceFile.Writer.replication((short)2), MapFile.Writer.keyClass(IntWritable.class), MapFile.Writer.valueClass(IntWritable.class) ); w.append(new IntWritable(20), new IntWritable(300)); w.append(new IntWritable(30), new IntWritable(400)); } catch (Exception ex) { ex.printStackTrace(); } finally { w.close(); } MapFile.Reader reader = new MapFile.Reader( new Path("hdfs://192.168.56.101:9000/20161020/data"), new Configuration() ); System.out.println(reader.get(new IntWritable(10), new IntWritable())); //print 200 System.out.println(reader.get(new IntWritable(20), new IntWritable())); //print 300 MapFile correct reader.close(); 3. Append to exist MapFile, but incorrect range MapFile.Writer w = null; try { w = new MapFile.Writer( new Configuration(), new Path("hdfs://192.168.56.101:9000/20161020/data"), MapFile.Writer.keyClass(IntWritable.class), MapFile.Writer.valueClass(IntWritable.class) ); w.append(new IntWritable(10), new IntWritable(100)); w.append(new IntWritable(20), new IntWritable(200)); w.close(); w = new MapFile.Writer( new Configuration(), new Path("hdfs://192.168.56.101:9000/20161020/data"), SequenceFile.Writer.appendIfExists(true), //append to MapFile MapFile.Writer.keyClass(IntWritable.class), MapFile.Writer.valueClass(IntWritable.class) ); w.append(new IntWritable(5), new IntWritable(300)); //No exception here w.append(new IntWritable(10), new IntWritable(400)); } catch (Exception ex) { ex.printStackTrace(); } finally { w.close(); } MapFile.Reader reader = new MapFile.Reader( new Path("hdfs://192.168.56.101:9000/20161020/data"), new Configuration() ); System.out.println(reader.get(new IntWritable(5), new IntWritable())); // java.io.IOException: key out of order: 5 after 10 - MapFile corrupted System.out.println(reader.get(new IntWritable(10), new IntWritable())); System.out.println(reader.get(new IntWritable(20), new IntWritable())); reader.close(); {code} Reason: When open MapFile with option SequenceFile.Writer.appendIfExists(true) - writer not read last key from exist MapFile. > Open MapFile for append > ----------------------- > > Key: HADOOP-13725 > URL: https://issues.apache.org/jira/browse/HADOOP-13725 > Project: Hadoop Common > Issue Type: New Feature > Reporter: VITALIY SAVCHENKO > > I think it possible to open MapFile for appending. > SequenceFile support it (Option SequenceFile.Writer.appendIfExists(true) > HADOOP-7139) > Now it almost working. But if use SequenceFile.Writer.appendIfExists(true) > MapFile.Writer - it not read last key and does not check new keys. That's why > MapFile can be corrupted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org