[jira] [Commented] (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
[ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061833#comment-13061833 ] Konstantin Shvachko commented on HDFS-1073: --- I would also like to ask for some benchmarks to make sure we do not loose in performance for NN operations. NNThorughtput is applicable in the case. But other tests are welcome as well. > Simpler model for Namenode's fs Image and edit Logs > > > Key: HDFS-1073 > URL: https://issues.apache.org/jira/browse/HDFS-1073 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sanjay Radia >Assignee: Todd Lipcon > Attachments: hdfs-1073-editloading-algos.txt, hdfs-1073.txt, > hdfs1073.pdf, hdfs1073.pdf, hdfs1073.pdf, hdfs1073.tex > > > The naming and handling of NN's fsImage and edit logs can be significantly > improved resulting simpler and more robust code. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2018) Move all journal stream management code into one place
[ https://issues.apache.org/jira/browse/HDFS-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061844#comment-13061844 ] Ivan Kelly commented on HDFS-2018: -- Good comments, I have a 3 hour flight later so I'll try to address them on it. A few points. Firstly, I'd really like to get this into 0.23. Once this patch is in, bookkeeper support can be added as a third party module with very little effort for 0.23, and it can be shipped as a builtin in later releases. The remoteEditLogManifest will only be used by file based journals. For bookkeeper the checkpointer will simply be configured to use bookkeeper as the editlog. GetImageServlet will be taken out of the equation for edits. Regarding the startup behaviour, I would need to add getFirstTxId() and getLastTxId() members to EditLogInputStream to be able to create a plan beforehand. Ill have a go at this later, and try to submit another patch once I get back near an internet connection. > Move all journal stream management code into one place > -- > > Key: HDFS-2018 > URL: https://issues.apache.org/jira/browse/HDFS-2018 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, > HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, > HDFS-2018.diff, HDFS-2018.diff > > > Currently in the HDFS-1073 branch, the code for creating output streams is in > FileJournalManager and the code for input streams is in the inspectors. This > change does a number of things. > - Input and Output streams are now created by the JournalManager. > - FSImageStorageInspectors now deals with URIs when referring to edit logs > - Recovery of inprogress logs is performed by counting the number of > transactions instead of looking at the length of the file. > The patch for this applies on top of the HDFS-1073 branch + HDFS-2003 patch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2111) Add tests for ensuring that the DN will start with a few bad data directories (Part 1 of testing DiskChecker)
[ https://issues.apache.org/jira/browse/HDFS-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061933#comment-13061933 ] Hudson commented on HDFS-2111: -- Integrated in Hadoop-Hdfs-trunk #719 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/719/]) HDFS-2111. Add tests for ensuring that the DN will start with a few bad data directories. Contributed by Harsh J Chouraria. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144100 Files : * /hadoop/common/trunk/hdfs/CHANGES.txt * /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java > Add tests for ensuring that the DN will start with a few bad data directories > (Part 1 of testing DiskChecker) > - > > Key: HDFS-2111 > URL: https://issues.apache.org/jira/browse/HDFS-2111 > Project: Hadoop HDFS > Issue Type: Test > Components: data-node, test >Affects Versions: 0.23.0 >Reporter: Harsh J >Assignee: Harsh J > Labels: test > Fix For: 0.23.0 > > Attachments: HDFS-2111.r1.diff, HDFS-2111.r1.diff > > > Add tests to ensure that given multiple data dirs, if a single is bad, the DN > should still start up. > This is to check DiskChecker's functionality used in instantiating DataNodes -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George updated HDFS-2034:
--
Attachment: HDFS-2034-5.patch
Daryn,
thank you for reviewing this patch again. I agree with you on changing the
assert to Exception. The attached patch has that changed.
I have left out checking for the offset > getFileLength() again since the
exception should cover for that case.
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034-4.patch, HDFS-2034-5.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2131) Tests for HADOOP-7361
[ https://issues.apache.org/jira/browse/HDFS-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061990#comment-13061990 ] Daryn Sharp commented on HDFS-2131: --- +1 good job! > Tests for HADOOP-7361 > - > > Key: HDFS-2131 > URL: https://issues.apache.org/jira/browse/HDFS-2131 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HADOOP-7361-test.patch, HADOOP-7361-test.patch, > HADOOP-7361-test.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1981) When namenode goes down while checkpointing and if is started again subsequent Checkpointing is always failing
[ https://issues.apache.org/jira/browse/HDFS-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HDFS-1981: - Status: Open (was: Patch Available) > When namenode goes down while checkpointing and if is started again > subsequent Checkpointing is always failing > -- > > Key: HDFS-1981 > URL: https://issues.apache.org/jira/browse/HDFS-1981 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 > Environment: Linux >Reporter: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.22.0 > > Attachments: HDFS-1981-1.patch, HDFS-1981-2.patch, HDFS-1981.patch > > > This scenario is applicable in NN and BNN case. > When the namenode goes down after creating the edits.new, on subsequent > restart the divertFileStreams will not happen to edits.new as the edits.new > file is already present and the size is zero. > so on trying to saveCheckPoint an exception occurs > 2011-05-23 16:38:57,476 WARN org.mortbay.log: /getimage: java.io.IOException: > GetImage failed. java.io.IOException: Namenode has an edit log with timestamp > of 2011-05-23 16:38:56 but new checkpoint was created using editlog with > timestamp 2011-05-23 16:37:30. Checkpoint Aborted. > This is a bug or is that the behaviour. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1981) When namenode goes down while checkpointing and if is started again subsequent Checkpointing is always failing
[ https://issues.apache.org/jira/browse/HDFS-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HDFS-1981: - Attachment: HDFS-1981-2.patch > When namenode goes down while checkpointing and if is started again > subsequent Checkpointing is always failing > -- > > Key: HDFS-1981 > URL: https://issues.apache.org/jira/browse/HDFS-1981 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 > Environment: Linux >Reporter: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.22.0 > > Attachments: HDFS-1981-1.patch, HDFS-1981-2.patch, HDFS-1981.patch > > > This scenario is applicable in NN and BNN case. > When the namenode goes down after creating the edits.new, on subsequent > restart the divertFileStreams will not happen to edits.new as the edits.new > file is already present and the size is zero. > so on trying to saveCheckPoint an exception occurs > 2011-05-23 16:38:57,476 WARN org.mortbay.log: /getimage: java.io.IOException: > GetImage failed. java.io.IOException: Namenode has an edit log with timestamp > of 2011-05-23 16:38:56 but new checkpoint was created using editlog with > timestamp 2011-05-23 16:37:30. Checkpoint Aborted. > This is a bug or is that the behaviour. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1981) When namenode goes down while checkpointing and if is started again subsequent Checkpointing is always failing
[ https://issues.apache.org/jira/browse/HDFS-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HDFS-1981: - Status: Patch Available (was: Open) > When namenode goes down while checkpointing and if is started again > subsequent Checkpointing is always failing > -- > > Key: HDFS-1981 > URL: https://issues.apache.org/jira/browse/HDFS-1981 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 > Environment: Linux >Reporter: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.22.0 > > Attachments: HDFS-1981-1.patch, HDFS-1981-2.patch, HDFS-1981.patch > > > This scenario is applicable in NN and BNN case. > When the namenode goes down after creating the edits.new, on subsequent > restart the divertFileStreams will not happen to edits.new as the edits.new > file is already present and the size is zero. > so on trying to saveCheckPoint an exception occurs > 2011-05-23 16:38:57,476 WARN org.mortbay.log: /getimage: java.io.IOException: > GetImage failed. java.io.IOException: Namenode has an edit log with timestamp > of 2011-05-23 16:38:56 but new checkpoint was created using editlog with > timestamp 2011-05-23 16:37:30. Checkpoint Aborted. > This is a bug or is that the behaviour. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1981) When namenode goes down while checkpointing and if is started again subsequent Checkpointing is always failing
[ https://issues.apache.org/jira/browse/HDFS-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062011#comment-13062011 ] ramkrishna.s.vasudevan commented on HDFS-1981: -- Hi Todd, Thanks for your comments.. I have reworked on some of the comments. I think you have reviewed the old patch and not the patch with the name HDFS-1981-1.patch Any way I have corrected some of the comments in the latest patch also * As Konstantin said, please use Junit 4 (annotations API) instead of Junit 3, and use the MiniDFSCluster builder Already Addressed in previous patch. * typo: NEW_EIDTS_STREAM have changed this to NEW_EDITS_STREAM * don't use the string constant "dfs.name.dir" - there are constants in DFSConfigKeys for this Updated * "false == editsNew.exists()" ?? !editsNew.exists() Udpated * TODOs in the test case. don't swallow exceptions Updated * you can use IOUtils.cleanup or IOUtils.closeStream in the finally block inside of the block Updated * no need to clear editsStreams in teardown method - it's an instance var so it will be recreated for each case anyway Updated * what's the purpose of the setup which creates bImg? It's not used in any of the test cases. Instead of using the variable bImg, have now created an instance local level * assertion text is wrong: "image should be deleted" – but it's checking that "edits.new" should be deleted. Fixed in the previous patch- as per the latest fix told by Konstantin > When namenode goes down while checkpointing and if is started again > subsequent Checkpointing is always failing > -- > > Key: HDFS-1981 > URL: https://issues.apache.org/jira/browse/HDFS-1981 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 > Environment: Linux >Reporter: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.22.0 > > Attachments: HDFS-1981-1.patch, HDFS-1981-2.patch, HDFS-1981.patch > > > This scenario is applicable in NN and BNN case. > When the namenode goes down after creating the edits.new, on subsequent > restart the divertFileStreams will not happen to edits.new as the edits.new > file is already present and the size is zero. > so on trying to saveCheckPoint an exception occurs > 2011-05-23 16:38:57,476 WARN org.mortbay.log: /getimage: java.io.IOException: > GetImage failed. java.io.IOException: Namenode has an edit log with timestamp > of 2011-05-23 16:38:56 but new checkpoint was created using editlog with > timestamp 2011-05-23 16:37:30. Checkpoint Aborted. > This is a bug or is that the behaviour. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062023#comment-13062023
]
Daryn Sharp commented on HDFS-2034:
---
+1 Great! Now we're protected whether or not asserts are enabled.
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034-4.patch, HDFS-2034-5.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062024#comment-13062024
]
Hadoop QA commented on HDFS-2034:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12485740/HDFS-2034-5.patch
against trunk revision 1144100.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestDFSShell
org.apache.hadoop.hdfs.TestHDFSTrash
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/899//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/899//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/899//console
This message is automatically generated.
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034-4.patch, HDFS-2034-5.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1977) Stop using StringUtils.stringifyException()
[ https://issues.apache.org/jira/browse/HDFS-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062068#comment-13062068 ] Bharath Mundlapudi commented on HDFS-1977: -- Todd, If you don't have any further comments on this patch, can you please commit this? > Stop using StringUtils.stringifyException() > --- > > Key: HDFS-1977 > URL: https://issues.apache.org/jira/browse/HDFS-1977 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Joey Echeverria >Assignee: Bharath Mundlapudi >Priority: Minor > Attachments: HDFS-1977-1.patch, HDFS-1977-2.patch, HDFS-1977-3.patch > > > The old version of the logging APIs didn't support logging stack traces by > passing exceptions to the logging methods (e.g. Log.error()). A number of log > statements make use of StringUtils.stringifyException() to get around the old > behavior. It would be nice if this could get cleaned up to make use of the > the logger's stack trace printing. This also gives users more control since > you can configure how the stack traces are written to the logs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2138) fix aop.xml to refer to the right hadoop-common.version variable
[ https://issues.apache.org/jira/browse/HDFS-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated HDFS-2138: - Attachment: HDFS-2138-trunk.patch Thanks Cos I was fixing this on the MR-279 branch and uploaded the patch without realizing that this would not apply for trunk. HDFS-2138-trunk.patch will work with trunk. Could you pls take a look? > fix aop.xml to refer to the right hadoop-common.version variable > > > Key: HDFS-2138 > URL: https://issues.apache.org/jira/browse/HDFS-2138 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 0.23.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Attachments: HDFS-2138-trunk.patch, HDFS-2138.PATCH > > > aop.xml refers to hadoop-common version through project.version variable; > Instead hadoop-common version should be referred through > hadoop-common.version set in ivy/libraries.properties file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2138) fix aop.xml to refer to the right hadoop-common.version variable
[ https://issues.apache.org/jira/browse/HDFS-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated HDFS-2138: - Status: Patch Available (was: Open) > fix aop.xml to refer to the right hadoop-common.version variable > > > Key: HDFS-2138 > URL: https://issues.apache.org/jira/browse/HDFS-2138 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 0.23.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Attachments: HDFS-2138-trunk.patch, HDFS-2138.PATCH > > > aop.xml refers to hadoop-common version through project.version variable; > Instead hadoop-common version should be referred through > hadoop-common.version set in ivy/libraries.properties file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2138) fix aop.xml to refer to the right hadoop-common.version variable
[ https://issues.apache.org/jira/browse/HDFS-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated HDFS-2138: - Status: Open (was: Patch Available) > fix aop.xml to refer to the right hadoop-common.version variable > > > Key: HDFS-2138 > URL: https://issues.apache.org/jira/browse/HDFS-2138 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 0.23.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Attachments: HDFS-2138-trunk.patch, HDFS-2138.PATCH > > > aop.xml refers to hadoop-common version through project.version variable; > Instead hadoop-common version should be referred through > hadoop-common.version set in ivy/libraries.properties file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2134) Move DecommissionManager to block management
[
https://issues.apache.org/jira/browse/HDFS-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062110#comment-13062110
]
Tsz Wo (Nicholas), SZE commented on HDFS-2134:
--
Todd, thanks for taking a look. The ordering of thread interrupt should not
cause any problem.
> Move DecommissionManager to block management
>
>
> Key: HDFS-2134
> URL: https://issues.apache.org/jira/browse/HDFS-2134
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2134_20110706.patch
>
>
> Datanode management including {{DecommissionManager}} should belong to block
> management.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2134) Move DecommissionManager to block management
[
https://issues.apache.org/jira/browse/HDFS-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062109#comment-13062109
]
Tsz Wo (Nicholas), SZE commented on HDFS-2134:
--
Todd, thanks for taking a look. The ordering of thread interrupt should not
cause any problem.
> Move DecommissionManager to block management
>
>
> Key: HDFS-2134
> URL: https://issues.apache.org/jira/browse/HDFS-2134
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2134_20110706.patch
>
>
> Datanode management including {{DecommissionManager}} should belong to block
> management.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2138) fix aop.xml to refer to the right hadoop-common.version variable
[ https://issues.apache.org/jira/browse/HDFS-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062124#comment-13062124 ] Hadoop QA commented on HDFS-2138: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485768/HDFS-2138-trunk.patch against trunk revision 1144100. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestHDFSTrash +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/901//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/901//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/901//console This message is automatically generated. > fix aop.xml to refer to the right hadoop-common.version variable > > > Key: HDFS-2138 > URL: https://issues.apache.org/jira/browse/HDFS-2138 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 0.23.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Attachments: HDFS-2138-trunk.patch, HDFS-2138.PATCH > > > aop.xml refers to hadoop-common version through project.version variable; > Instead hadoop-common version should be referred through > hadoop-common.version set in ivy/libraries.properties file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2134) Move DecommissionManager to block management
[
https://issues.apache.org/jira/browse/HDFS-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2134:
-
Attachment: h2134_20110708.patch
h2134_20110708.patch: forgot to call {{DatanodeManager.activate(..)}}.
> Move DecommissionManager to block management
>
>
> Key: HDFS-2134
> URL: https://issues.apache.org/jira/browse/HDFS-2134
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2134_20110706.patch, h2134_20110708.patch
>
>
> Datanode management including {{DecommissionManager}} should belong to block
> management.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2134) Move DecommissionManager to block management
[
https://issues.apache.org/jira/browse/HDFS-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062129#comment-13062129
]
Hadoop QA commented on HDFS-2134:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12485523/h2134_20110706.patch
against trunk revision 1143147.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/888//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/888//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/888//console
This message is automatically generated.
> Move DecommissionManager to block management
>
>
> Key: HDFS-2134
> URL: https://issues.apache.org/jira/browse/HDFS-2134
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2134_20110706.patch, h2134_20110708.patch
>
>
> Datanode management including {{DecommissionManager}} should belong to block
> management.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2134) Move DecommissionManager to block management
[
https://issues.apache.org/jira/browse/HDFS-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062130#comment-13062130
]
Hadoop QA commented on HDFS-2134:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12485775/h2134_20110708.patch
against trunk revision 1144100.
-1 @author. The patch appears to contain @author tags which the Hadoop
community has agreed to not allow in code contributions.
+1 tests included. The patch appears to include new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/902//console
This message is automatically generated.
> Move DecommissionManager to block management
>
>
> Key: HDFS-2134
> URL: https://issues.apache.org/jira/browse/HDFS-2134
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2134_20110706.patch, h2134_20110708.patch
>
>
> Datanode management including {{DecommissionManager}} should belong to block
> management.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062133#comment-13062133
]
John George commented on HDFS-2034:
---
I don't think the above tests fail because of this patch, but I dont see the
tests failing in the patches before this one... When I run these tests
manually, they seem to pass as well.
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034-4.patch, HDFS-2034-5.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2134) Move DecommissionManager to block management
[
https://issues.apache.org/jira/browse/HDFS-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062136#comment-13062136
]
Tsz Wo (Nicholas), SZE commented on HDFS-2134:
--
bq. -1 @author. The patch appears to contain @author tags which the Hadoop
community has agreed to not allow in code contributions.
There is no author tags in the patch. Something wrong with hudson?!
> Move DecommissionManager to block management
>
>
> Key: HDFS-2134
> URL: https://issues.apache.org/jira/browse/HDFS-2134
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2134_20110706.patch, h2134_20110708.patch
>
>
> Datanode management including {{DecommissionManager}} should belong to block
> management.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062145#comment-13062145
]
Tsz Wo (Nicholas), SZE commented on HDFS-2034:
--
I agreed that the failed tests are not related. Some tests failed with
{noformat}
java.lang.IllegalArgumentException: port out of range:-1
at java.net.InetSocketAddress.(InetSocketAddress.java:118)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$1.run(NameNode.java:597)
...
{noformat}
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034-4.patch, HDFS-2034-5.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2034:
-
Resolution: Fixed
Fix Version/s: 0.23.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I have committed this. Thanks, John.
Also thanks Daryn for reviewing it.
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034-4.patch, HDFS-2034-5.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2034:
-
Component/s: hdfs client
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034-4.patch, HDFS-2034-5.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062163#comment-13062163
]
Hudson commented on HDFS-2034:
--
Integrated in Hadoop-Hdfs-trunk-Commit #776 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/776/])
HDFS-2034. Length in DFSInputStream.getBlockRange(..) becomes -ve when
reading only from a currently being written block. Contributed by John George
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1144480
Files :
* /hadoop/common/trunk/hdfs/CHANGES.txt
* /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/DFSInputStream.java
*
/hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestWriteRead.java
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034-4.patch, HDFS-2034-5.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
[ https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDFS-1975: -- Assignee: Jitendra Nath Pandey > HA: Support for sharing the namenode state from active to standby. > -- > > Key: HDFS-1975 > URL: https://issues.apache.org/jira/browse/HDFS-1975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Suresh Srinivas >Assignee: Jitendra Nath Pandey > > To enable hot standby namenode, the standby node must have current > information for - namenode state (image + edits) and block location > information. This jira addresses keeping the namenode state current in the > standby node. To do this, the proposed solution in this jira is to use a > shared storage to store the namenode state. > Note one could also build an alternative solution by augmenting the backup > node. A seperate jira could explore this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2134) Move DecommissionManager to block management
[
https://issues.apache.org/jira/browse/HDFS-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062198#comment-13062198
]
Hadoop QA commented on HDFS-2134:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12485775/h2134_20110708.patch
against trunk revision 1144480.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestHDFSTrash
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/903//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/903//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/903//console
This message is automatically generated.
> Move DecommissionManager to block management
>
>
> Key: HDFS-2134
> URL: https://issues.apache.org/jira/browse/HDFS-2134
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2134_20110706.patch, h2134_20110708.patch
>
>
> Datanode management including {{DecommissionManager}} should belong to block
> management.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2138) fix aop.xml to refer to the right hadoop-common.version variable
[ https://issues.apache.org/jira/browse/HDFS-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062236#comment-13062236 ] Matt Foley commented on HDFS-2138: -- +1. lgtm. The failure of unit test TestTrash.testTrashEmptier does not seem to be related to this change. It is probably related to HDFS-7326, although the symptoms reported are slightly different. > fix aop.xml to refer to the right hadoop-common.version variable > > > Key: HDFS-2138 > URL: https://issues.apache.org/jira/browse/HDFS-2138 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 0.23.0 >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Attachments: HDFS-2138-trunk.patch, HDFS-2138.PATCH > > > aop.xml refers to hadoop-common version through project.version variable; > Instead hadoop-common version should be referred through > hadoop-common.version set in ivy/libraries.properties file. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2131) Tests for HADOOP-7361
[ https://issues.apache.org/jira/browse/HDFS-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2131: -- Component/s: test > Tests for HADOOP-7361 > - > > Key: HDFS-2131 > URL: https://issues.apache.org/jira/browse/HDFS-2131 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HADOOP-7361-test.patch, HADOOP-7361-test.patch, > HADOOP-7361-test.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2131) Tests for HADOOP-7361
[ https://issues.apache.org/jira/browse/HDFS-2131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062246#comment-13062246 ] Uma Maheswara Rao G commented on HDFS-2131: --- Thanks Daryn! for taking a look. > Tests for HADOOP-7361 > - > > Key: HDFS-2131 > URL: https://issues.apache.org/jira/browse/HDFS-2131 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HADOOP-7361-test.patch, HADOOP-7361-test.patch, > HADOOP-7361-test.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1971) HA: Send block report from datanode to both active and standby namenodes
[ https://issues.apache.org/jira/browse/HDFS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062250#comment-13062250 ] Uma Maheswara Rao G commented on HDFS-1971: --- Hi Sanjay, Are you started working on this? If not, can i take this issue? Because in our cluster, we already implemented this mechanism. So, we will be happy to contribute our efforts :-) > HA: Send block report from datanode to both active and standby namenodes > > > Key: HDFS-1971 > URL: https://issues.apache.org/jira/browse/HDFS-1971 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, name-node >Reporter: Suresh Srinivas >Assignee: Sanjay Radia > > To enable hot standby namenode, the standby node must have current > information for - namenode state (image + edits) and block location > information. This jira addresses keeping the block location information > current in the standby node. To do this, the proposed solution is to send > block reports from the datanodes to both the active and the standby namenode. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2071) Use of isConnected() in DataXceiver is invalid
[
https://issues.apache.org/jira/browse/HDFS-2071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062251#comment-13062251
]
Hudson commented on HDFS-2071:
--
Integrated in Hadoop-Hdfs-22-branch #70 (See
[https://builds.apache.org/job/Hadoop-Hdfs-22-branch/70/])
> Use of isConnected() in DataXceiver is invalid
> --
>
> Key: HDFS-2071
> URL: https://issues.apache.org/jira/browse/HDFS-2071
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Minor
> Fix For: 0.22.0
>
> Attachments: HDFS-2071.patch
>
>
> The use of Socket.isConnected() in DataXceiver.run() is not valid. It returns
> false until the connection is made and then always returns true after that.
> It will never return false after the initial connection is successfully made.
> Socket.isClosed() or SocketChannel.isOpen() should be used instead, assuming
> someone is handling SocketException and does Socket.close() or
> SocketChannel.close(). It seems the op handlers in DataXceiver are diligently
> using IOUtils.closeStream(), which will invoke SocketChannel.close().
> {code}
> - } while (s.isConnected() && socketKeepaliveTimeout > 0);
> + } while (!s.isClosed() && socketKeepaliveTimeout > 0);
> {code}
> The effect of this bug is very minor, as the socket is read again right
> after. If the connection was closed, the readOp() will throw an EOFException,
> which is caught and dealt with properly. The system still functions normally
> with probably only few microseconds of extra overhead in the premature
> connection closure cases.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
[ https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062253#comment-13062253 ] Hudson commented on HDFS-941: - Integrated in Hadoop-Hdfs-22-branch #70 (See [https://builds.apache.org/job/Hadoop-Hdfs-22-branch/70/]) > Datanode xceiver protocol should allow reuse of a connection > > > Key: HDFS-941 > URL: https://issues.apache.org/jira/browse/HDFS-941 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, hdfs client >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: bc Wong > Fix For: 0.22.0 > > Attachments: 941.22.txt, 941.22.txt, 941.22.v2.txt, 941.22.v3.txt, > HDFS-941-1.patch, HDFS-941-2.patch, HDFS-941-3.patch, HDFS-941-3.patch, > HDFS-941-4.patch, HDFS-941-5.patch, HDFS-941-6.22.patch, HDFS-941-6.patch, > HDFS-941-6.patch, HDFS-941-6.patch, fix-close-delta.txt, hdfs-941.txt, > hdfs-941.txt, hdfs-941.txt, hdfs-941.txt, hdfs941-1.png > > > Right now each connection into the datanode xceiver only processes one > operation. > In the case that an operation leaves the stream in a well-defined state (eg a > client reads to the end of a block successfully) the same connection could be > reused for a second operation. This should improve random read performance > significantly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1952) FSEditLog.open() appears to succeed even if all EDITS directories fail
[ https://issues.apache.org/jira/browse/HDFS-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062252#comment-13062252 ] Hudson commented on HDFS-1952: -- Integrated in Hadoop-Hdfs-22-branch #70 (See [https://builds.apache.org/job/Hadoop-Hdfs-22-branch/70/]) > FSEditLog.open() appears to succeed even if all EDITS directories fail > -- > > Key: HDFS-1952 > URL: https://issues.apache.org/jira/browse/HDFS-1952 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0, 0.23.0 >Reporter: Matt Foley >Assignee: Andrew Wang > Labels: newbie > Fix For: 0.22.0, 0.23.0 > > Attachments: hdfs-1952-0.22.patch, hdfs-1952.patch, hdfs-1952.patch, > hdfs-1952.patch > > > FSEditLog.open() appears to "succeed" even if all of the individual > directories failed to allow creation of an EditLogOutputStream. The problem > and solution are essentially similar to that of HDFS-1505. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
