[jira] [Updated] (HDFS-8964) When validating the edit log, do not read at or beyond the file offset that is being written

2015-09-03 Thread Colin Patrick McCabe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HDFS-8964:
---
Description: 
NN/JN validates in-progress edit log files in multiple scenarios, via 
{{EditLogFile#validateLog}}. The method scans through the edit log file to find 
the last transaction ID.

However, an in-progress edit log file could be actively written to, which 
creates a race condition and causes incorrect data to be read (and later we 
attempt to interpret the data as ops).  This causes problems for INotify, which 
reads edit log entries while the edit log is still being written.

Currently {{validateLog}} is used in 3 places:
# NN {{getEditsFromTxid}}
# JN {{getEditLogManifest}}
# NN/JN {{recoverUnfinalizedSegments}}

In the first two scenarios we should provide a maximum TxId to validate in the 
in-progress file. The 3rd scenario won't cause a race condition because only 
non-current in-progress edit log files are validated.

{{validateLog}} is actually only used with in-progress files, and could use a 
better name and Javadoc.

  was:
NN/JN validates in-progress edit log files in multiple scenarios, via 
{{EditLogFile#validateLog}}. The method scans through the edit log file to find 
the last transaction ID.

However, an in-progress edit log file could be actively written to, which 
creates a race condition and causes incorrect data to be read (and later we 
attempt to interpret the data as ops).

Currently {{validateLog}} is used in 3 places:
# NN {{getEditsFromTxid}}
# JN {{getEditLogManifest}}
# NN/JN {{recoverUnfinalizedSegments}}

In the first two scenarios we should provide a maximum TxId to validate in the 
in-progress file. The 3rd scenario won't cause a race condition because only 
non-current in-progress edit log files are validated.

{{validateLog}} is actually only used with in-progress files, and could use a 
better name and Javadoc.


> When validating the edit log, do not read at or beyond the file offset that 
> is being written
> 
>
> Key: HDFS-8964
> URL: https://issues.apache.org/jira/browse/HDFS-8964
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: journal-node, namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8964.00.patch, HDFS-8964.01.patch, 
> HDFS-8964.02.patch, HDFS-8964.03.patch, HDFS-8964.04.patch, 
> HDFS-8964.05.patch, HDFS-8964.06.patch
>
>
> NN/JN validates in-progress edit log files in multiple scenarios, via 
> {{EditLogFile#validateLog}}. The method scans through the edit log file to 
> find the last transaction ID.
> However, an in-progress edit log file could be actively written to, which 
> creates a race condition and causes incorrect data to be read (and later we 
> attempt to interpret the data as ops).  This causes problems for INotify, 
> which reads edit log entries while the edit log is still being written.
> Currently {{validateLog}} is used in 3 places:
> # NN {{getEditsFromTxid}}
> # JN {{getEditLogManifest}}
> # NN/JN {{recoverUnfinalizedSegments}}
> In the first two scenarios we should provide a maximum TxId to validate in 
> the in-progress file. The 3rd scenario won't cause a race condition because 
> only non-current in-progress edit log files are validated.
> {{validateLog}} is actually only used with in-progress files, and could use a 
> better name and Javadoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-8964) When validating the edit log, do not read at or beyond the file offset that is being written

2015-09-03 Thread Colin Patrick McCabe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HDFS-8964:
---
Summary: When validating the edit log, do not read at or beyond the file 
offset that is being written  (was: INotify should not read at or beyond the 
file offset that is being written in FileJournalManager)

> When validating the edit log, do not read at or beyond the file offset that 
> is being written
> 
>
> Key: HDFS-8964
> URL: https://issues.apache.org/jira/browse/HDFS-8964
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: journal-node, namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-8964.00.patch, HDFS-8964.01.patch, 
> HDFS-8964.02.patch, HDFS-8964.03.patch, HDFS-8964.04.patch, 
> HDFS-8964.05.patch, HDFS-8964.06.patch
>
>
> NN/JN validates in-progress edit log files in multiple scenarios, via 
> {{EditLogFile#validateLog}}. The method scans through the edit log file to 
> find the last transaction ID.
> However, an in-progress edit log file could be actively written to, which 
> creates a race condition and causes incorrect data to be read (and later we 
> attempt to interpret the data as ops).
> Currently {{validateLog}} is used in 3 places:
> # NN {{getEditsFromTxid}}
> # JN {{getEditLogManifest}}
> # NN/JN {{recoverUnfinalizedSegments}}
> In the first two scenarios we should provide a maximum TxId to validate in 
> the in-progress file. The 3rd scenario won't cause a race condition because 
> only non-current in-progress edit log files are validated.
> {{validateLog}} is actually only used with in-progress files, and could use a 
> better name and Javadoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-8964) When validating the edit log, do not read at or beyond the file offset that is being written

2015-09-03 Thread Colin Patrick McCabe (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HDFS-8964:
---
   Resolution: Fixed
Fix Version/s: 2.8
   Status: Resolved  (was: Patch Available)

> When validating the edit log, do not read at or beyond the file offset that 
> is being written
> 
>
> Key: HDFS-8964
> URL: https://issues.apache.org/jira/browse/HDFS-8964
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: journal-node, namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8
>
> Attachments: HDFS-8964.00.patch, HDFS-8964.01.patch, 
> HDFS-8964.02.patch, HDFS-8964.03.patch, HDFS-8964.04.patch, 
> HDFS-8964.05.patch, HDFS-8964.06.patch
>
>
> NN/JN validates in-progress edit log files in multiple scenarios, via 
> {{EditLogFile#validateLog}}. The method scans through the edit log file to 
> find the last transaction ID.
> However, an in-progress edit log file could be actively written to, which 
> creates a race condition and causes incorrect data to be read (and later we 
> attempt to interpret the data as ops).  This causes problems for INotify, 
> which reads edit log entries while the edit log is still being written.
> Currently {{validateLog}} is used in 3 places:
> # NN {{getEditsFromTxid}}
> # JN {{getEditLogManifest}}
> # NN/JN {{recoverUnfinalizedSegments}}
> In the first two scenarios we should provide a maximum TxId to validate in 
> the in-progress file. The 3rd scenario won't cause a race condition because 
> only non-current in-progress edit log files are validated.
> {{validateLog}} is actually only used with in-progress files, and could use a 
> better name and Javadoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HDFS-8964) When validating the edit log, do not read at or beyond the file offset that is being written

2015-09-03 Thread Allen Wittenauer (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allen Wittenauer updated HDFS-8964:
---
Fix Version/s: (was: 2.8)
   2.8.0

> When validating the edit log, do not read at or beyond the file offset that 
> is being written
> 
>
> Key: HDFS-8964
> URL: https://issues.apache.org/jira/browse/HDFS-8964
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: journal-node, namenode
>Affects Versions: 2.7.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0
>
> Attachments: HDFS-8964.00.patch, HDFS-8964.01.patch, 
> HDFS-8964.02.patch, HDFS-8964.03.patch, HDFS-8964.04.patch, 
> HDFS-8964.05.patch, HDFS-8964.06.patch
>
>
> NN/JN validates in-progress edit log files in multiple scenarios, via 
> {{EditLogFile#validateLog}}. The method scans through the edit log file to 
> find the last transaction ID.
> However, an in-progress edit log file could be actively written to, which 
> creates a race condition and causes incorrect data to be read (and later we 
> attempt to interpret the data as ops).  This causes problems for INotify, 
> which reads edit log entries while the edit log is still being written.
> Currently {{validateLog}} is used in 3 places:
> # NN {{getEditsFromTxid}}
> # JN {{getEditLogManifest}}
> # NN/JN {{recoverUnfinalizedSegments}}
> In the first two scenarios we should provide a maximum TxId to validate in 
> the in-progress file. The 3rd scenario won't cause a race condition because 
> only non-current in-progress edit log files are validated.
> {{validateLog}} is actually only used with in-progress files, and could use a 
> better name and Javadoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)