[ https://issues.apache.org/jira/browse/HDFS-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942461#comment-13942461 ]
Jing Zhao commented on HDFS-6038: --------------------------------- Thanks for the review, Nicholas! I will commit the patch shortly. > Allow JournalNode to handle editlog produced by new release with future > layoutversion > ------------------------------------------------------------------------------------- > > Key: HDFS-6038 > URL: https://issues.apache.org/jira/browse/HDFS-6038 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: journal-node, namenode > Reporter: Haohui Mai > Assignee: Jing Zhao > Attachments: HDFS-6038.000.patch, HDFS-6038.001.patch, > HDFS-6038.002.patch, HDFS-6038.003.patch, HDFS-6038.004.patch, > HDFS-6038.005.patch, HDFS-6038.006.patch, HDFS-6038.007.patch, > HDFS-6038.008.patch, editsStored > > > In HA setup, the JNs receive edit logs (blob) from the NN and write into edit > log files. In order to write well-formed edit log files, the JNs prepend a > header for each edit log file. The problem is that the JN hard-codes the > version (i.e., {{NameNodeLayoutVersion}} in the edit log, therefore it > generates incorrect edit logs when the newer release bumps the > {{NameNodeLayoutVersion}} during rolling upgrade. > In the meanwhile, currently JN tries to decode the in-progress editlog > segment in order to know the last txid in the segment. In the rolling upgrade > scenario, the JN with the old software may not be able to correctly decode > the editlog generated by the new software. > This jira makes the following changes to allow JN to handle editlog produced > by software with future layoutversion: > 1. Change the NN--JN startLogSegment RPC signature and let NN specify the > layoutversion for the new editlog segment. > 2. Persist a length field for each editlog op to indicate the total length of > the op. Instead of calling EditLogFileInputStream#validateEditLog to get the > last txid of an in-progress editlog segment, a new method scanEditLog is > added and used by JN which does not decode each editlog op but uses the > length to quickly jump to the next op. -- This message was sent by Atlassian JIRA (v6.2#6252)