[jira] [Updated] (HDFS-6038) Allow JournalNode to handle editlog produced by new release with future layoutversion
[
https://issues.apache.org/jira/browse/HDFS-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6038:
Summary: Allow JournalNode to handle editlog produced by new release with
future layoutversion (was: JournalNode hardcodes NameNodeLayoutVersion in the
edit log file)
> Allow JournalNode to handle editlog produced by new release with future
> layoutversion
> -
>
> Key: HDFS-6038
> URL: https://issues.apache.org/jira/browse/HDFS-6038
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: journal-node, namenode
>Reporter: Haohui Mai
>Assignee: Jing Zhao
> Attachments: HDFS-6038.000.patch, HDFS-6038.001.patch,
> HDFS-6038.002.patch, HDFS-6038.003.patch, HDFS-6038.004.patch,
> HDFS-6038.005.patch, HDFS-6038.006.patch, HDFS-6038.007.patch, editsStored
>
>
> In HA setup, the JNs receive edit logs (blob) from the NN and write into edit
> log files. In order to write well-formed edit log files, the JNs prepend a
> header for each edit log file. The problem is that the JN hard-codes the
> version (i.e., {{NameNodeLayoutVersion}} in the edit log, therefore it
> generates incorrect edit logs when the newer release bumps the
> {{NameNodeLayoutVersion}} during rolling upgrade.
> In the meanwhile, currently JN tries to decode the in-progress editlog
> segment in order to know the last txid in the segment. In the rolling upgrade
> scenario, the JN with the old software may not be able to correctly decode
> the editlog generated by the new software.
> This jira makes the following changes to allow JN to handle editlog produced
> by software with future layoutversion:
> 1. Change the NN--JN startLogSegment RPC signature and let NN specify the
> layoutversion for the new editlog segment.
> 2. Persist a length field for each editlog op to indicate the total length of
> the op. Instead of calling EditLogFileInputStream#validateEditLog to get the
> last txid of an in-progress editlog segment, a new method scanEditLog is
> added and used by JN which does not decode each editlog op but uses the
> length to quickly jump to the next op.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6038) Allow JournalNode to handle editlog produced by new release with future layoutversion
[
https://issues.apache.org/jira/browse/HDFS-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6038:
Attachment: HDFS-6038.008.patch
Update the patch to address Todd's comments. The main change is to add a new
unit test in TestJournal. In the new test we writes some editlog that JNs
cannot decode, and verifies that the JN can utilize the length field to scan
the segment.
> Allow JournalNode to handle editlog produced by new release with future
> layoutversion
> -
>
> Key: HDFS-6038
> URL: https://issues.apache.org/jira/browse/HDFS-6038
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: journal-node, namenode
>Reporter: Haohui Mai
>Assignee: Jing Zhao
> Attachments: HDFS-6038.000.patch, HDFS-6038.001.patch,
> HDFS-6038.002.patch, HDFS-6038.003.patch, HDFS-6038.004.patch,
> HDFS-6038.005.patch, HDFS-6038.006.patch, HDFS-6038.007.patch,
> HDFS-6038.008.patch, editsStored
>
>
> In HA setup, the JNs receive edit logs (blob) from the NN and write into edit
> log files. In order to write well-formed edit log files, the JNs prepend a
> header for each edit log file. The problem is that the JN hard-codes the
> version (i.e., {{NameNodeLayoutVersion}} in the edit log, therefore it
> generates incorrect edit logs when the newer release bumps the
> {{NameNodeLayoutVersion}} during rolling upgrade.
> In the meanwhile, currently JN tries to decode the in-progress editlog
> segment in order to know the last txid in the segment. In the rolling upgrade
> scenario, the JN with the old software may not be able to correctly decode
> the editlog generated by the new software.
> This jira makes the following changes to allow JN to handle editlog produced
> by software with future layoutversion:
> 1. Change the NN--JN startLogSegment RPC signature and let NN specify the
> layoutversion for the new editlog segment.
> 2. Persist a length field for each editlog op to indicate the total length of
> the op. Instead of calling EditLogFileInputStream#validateEditLog to get the
> last txid of an in-progress editlog segment, a new method scanEditLog is
> added and used by JN which does not decode each editlog op but uses the
> length to quickly jump to the next op.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6038) Allow JournalNode to handle editlog produced by new release with future layoutversion
[
https://issues.apache.org/jira/browse/HDFS-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6038:
Status: Patch Available (was: Open)
> Allow JournalNode to handle editlog produced by new release with future
> layoutversion
> -
>
> Key: HDFS-6038
> URL: https://issues.apache.org/jira/browse/HDFS-6038
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: journal-node, namenode
>Reporter: Haohui Mai
>Assignee: Jing Zhao
> Attachments: HDFS-6038.000.patch, HDFS-6038.001.patch,
> HDFS-6038.002.patch, HDFS-6038.003.patch, HDFS-6038.004.patch,
> HDFS-6038.005.patch, HDFS-6038.006.patch, HDFS-6038.007.patch,
> HDFS-6038.008.patch, editsStored
>
>
> In HA setup, the JNs receive edit logs (blob) from the NN and write into edit
> log files. In order to write well-formed edit log files, the JNs prepend a
> header for each edit log file. The problem is that the JN hard-codes the
> version (i.e., {{NameNodeLayoutVersion}} in the edit log, therefore it
> generates incorrect edit logs when the newer release bumps the
> {{NameNodeLayoutVersion}} during rolling upgrade.
> In the meanwhile, currently JN tries to decode the in-progress editlog
> segment in order to know the last txid in the segment. In the rolling upgrade
> scenario, the JN with the old software may not be able to correctly decode
> the editlog generated by the new software.
> This jira makes the following changes to allow JN to handle editlog produced
> by software with future layoutversion:
> 1. Change the NN--JN startLogSegment RPC signature and let NN specify the
> layoutversion for the new editlog segment.
> 2. Persist a length field for each editlog op to indicate the total length of
> the op. Instead of calling EditLogFileInputStream#validateEditLog to get the
> last txid of an in-progress editlog segment, a new method scanEditLog is
> added and used by JN which does not decode each editlog op but uses the
> length to quickly jump to the next op.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6038) Allow JournalNode to handle editlog produced by new release with future layoutversion
[
https://issues.apache.org/jira/browse/HDFS-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6038:
Resolution: Fixed
Fix Version/s: 2.4.0
Status: Resolved (was: Patch Available)
I've committed the patch to trunk, branch-2, and branch-2.4. Thanks Nicholas
and Todd for the review!
> Allow JournalNode to handle editlog produced by new release with future
> layoutversion
> -
>
> Key: HDFS-6038
> URL: https://issues.apache.org/jira/browse/HDFS-6038
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: journal-node, namenode
>Reporter: Haohui Mai
>Assignee: Jing Zhao
> Fix For: 2.4.0
>
> Attachments: HDFS-6038.000.patch, HDFS-6038.001.patch,
> HDFS-6038.002.patch, HDFS-6038.003.patch, HDFS-6038.004.patch,
> HDFS-6038.005.patch, HDFS-6038.006.patch, HDFS-6038.007.patch,
> HDFS-6038.008.patch, editsStored
>
>
> In HA setup, the JNs receive edit logs (blob) from the NN and write into edit
> log files. In order to write well-formed edit log files, the JNs prepend a
> header for each edit log file. The problem is that the JN hard-codes the
> version (i.e., {{NameNodeLayoutVersion}} in the edit log, therefore it
> generates incorrect edit logs when the newer release bumps the
> {{NameNodeLayoutVersion}} during rolling upgrade.
> In the meanwhile, currently JN tries to decode the in-progress editlog
> segment in order to know the last txid in the segment. In the rolling upgrade
> scenario, the JN with the old software may not be able to correctly decode
> the editlog generated by the new software.
> This jira makes the following changes to allow JN to handle editlog produced
> by software with future layoutversion:
> 1. Change the NN--JN startLogSegment RPC signature and let NN specify the
> layoutversion for the new editlog segment.
> 2. Persist a length field for each editlog op to indicate the total length of
> the op. Instead of calling EditLogFileInputStream#validateEditLog to get the
> last txid of an in-progress editlog segment, a new method scanEditLog is
> added and used by JN which does not decode each editlog op but uses the
> length to quickly jump to the next op.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
