[jira] [Resolved] (HDFS-2616) Change DatanodeProtocol#sendHeartbeat to return HeartbeatResponse
[ https://issues.apache.org/jira/browse/HDFS-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-2616. --- Resolution: Fixed Hadoop Flags: Reviewed Committed it to HDFS-1623 branch. Change DatanodeProtocol#sendHeartbeat to return HeartbeatResponse - Key: HDFS-2616 URL: https://issues.apache.org/jira/browse/HDFS-2616 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, name-node Affects Versions: HA branch (HDFS-1623) Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: HA branch (HDFS-1623) Attachments: HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt DatanodeProtocol#sendHeartbeat() returns DatanodeCommand[]. This jira proposes changing it to to return HeartbeatResponse that has DatanodeCommand[]. This allows adding other information that can be returned by the namenode to the datanode, instead of having to only return DatanodeCommand[]. For relevant discussion see HDFS-1972. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2618) Implement protobuf service for NamenodeProtocol
Implement protobuf service for NamenodeProtocol --- Key: HDFS-2618 URL: https://issues.apache.org/jira/browse/HDFS-2618 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 This jira adds implementation for NamenodeProtocol protobuf service along the same lines as HDFS-2181 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Jenkins build is unstable: Hadoop-Hdfs-HAbranch-build #2
See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/2/changes
Hadoop-Hdfs-HAbranch-build - Build # 2 - Unstable
See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/2/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 12521 lines...] [INFO] --- maven-javadoc-plugin:2.7:jar (module-javadocs) @ hadoop-hdfs-project --- [INFO] Not executing Javadoc as the project is not a Java classpath-capable package [INFO] [INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ hadoop-hdfs-project --- [INFO] Installing /home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-HAbranch-build/trunk/hadoop-hdfs-project/pom.xml to /home/jenkins/.m2/repository/org/apache/hadoop/hadoop-hdfs-project/0.23.1-SNAPSHOT/hadoop-hdfs-project-0.23.1-SNAPSHOT.pom [INFO] [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-hdfs-project --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-javadoc-plugin:2.7:jar (module-javadocs) @ hadoop-hdfs-project --- [INFO] Not executing Javadoc as the project is not a Java classpath-capable package [INFO] [INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ hadoop-hdfs-project --- [INFO] [INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ hadoop-hdfs-project --- [INFO] [INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ hadoop-hdfs-project --- [INFO] ** FindBugsMojo execute *** [INFO] canGenerate is false [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop HDFS SUCCESS [6:39.223s] [INFO] Apache Hadoop HDFS Project SUCCESS [0.120s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 6:39.688s [INFO] Finished at: Thu Dec 01 11:41:29 UTC 2011 [INFO] Final Memory: 62M/771M [INFO] + /home/jenkins/tools/maven/latest/bin/mvn test -Dmaven.test.failure.ignore=true -Pclover -DcloverLicenseLocation=/home/jenkins/tools/clover/latest/lib/clover.license Archiving artifacts Publishing Clover coverage report... Publishing Clover HTML report... Publishing Clover XML report... Publishing Clover coverage results... Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE Publishing Javadoc Recording fingerprints Updating HADOOP-7854 Updating MAPREDUCE-3488 Updating MAPREDUCE-3463 Updating MAPREDUCE-3477 Updating MAPREDUCE-3452 Updating HADOOP-7853 Sending e-mails to: hdfs-dev@hadoop.apache.org Email was triggered for: Unstable Sending email for trigger: Unstable ### ## FAILED TESTS (if any) ## 1 tests failed. REGRESSION: org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade.testDistributedUpgrade Error Message: test timed out after 12 milliseconds Stack Trace: java.lang.Exception: test timed out after 12 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:717) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:569) at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:261) at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:85) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:247) at org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade.__CLR3_0_2lxfdoy1dh2(TestDistributedUpgrade.java:139) at org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade.testDistributedUpgrade(TestDistributedUpgrade.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at
[jira] [Created] (HDFS-2619) Remove my personal email address from the libhdfs build file.
Remove my personal email address from the libhdfs build file.
-
Key: HDFS-2619
URL: https://issues.apache.org/jira/browse/HDFS-2619
Project: Hadoop HDFS
Issue Type: Bug
Components: build
Reporter: Owen O'Malley
Assignee: Owen O'Malley
When I first wrote the libhdfs autoconf/automake stuff, I incorrectly put my
email address in the AC_INIT line, which means if something goes wrong, you get:
{quote}
configure: WARNING: ## Report this to my-email ##
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2620) commands_manual.html balancer documentation missing -policy option
commands_manual.html balancer documentation missing -policy option -- Key: HDFS-2620 URL: https://issues.apache.org/jira/browse/HDFS-2620 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 0.23.0 Reporter: Thomas Graves the documentation for the balancer: commands_manual.html#balancer is missing the -policy option that was added with federation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2621) Balancer is not checking ALREADY_RUNNING state and never returns this state.
Balancer is not checking ALREADY_RUNNING state and never returns this state. Key: HDFS-2621 URL: https://issues.apache.org/jira/browse/HDFS-2621 Project: Hadoop HDFS Issue Type: Bug Reporter: Uma Maheswara Rao G -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2620) commands_manual.html balancer documentation missing -policy option
[ https://issues.apache.org/jira/browse/HDFS-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved HDFS-2620. - Resolution: Duplicate duplicate of HDFS-1685 commands_manual.html balancer documentation missing -policy option -- Key: HDFS-2620 URL: https://issues.apache.org/jira/browse/HDFS-2620 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 0.23.0 Reporter: Thomas Graves the documentation for the balancer: commands_manual.html#balancer is missing the -policy option that was added with federation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2622) HA: fix TestDFSUpgrade on HA branch
HA: fix TestDFSUpgrade on HA branch --- Key: HDFS-2622 URL: https://issues.apache.org/jira/browse/HDFS-2622 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, test Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Since HDFS-1975, we now increment the generation stamp for each block allocation. So the EXPECTED_TXID constant is wrong now in TestDFSUpgrade, since we do twice as many txns while creating files in the first phase of the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2622) HA: fix TestDFSUpgrade on HA branch
[ https://issues.apache.org/jira/browse/HDFS-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-2622. --- Resolution: Fixed Fix Version/s: HA branch (HDFS-1623) Hadoop Flags: Reviewed Committed, thanks Eli. HA: fix TestDFSUpgrade on HA branch --- Key: HDFS-2622 URL: https://issues.apache.org/jira/browse/HDFS-2622 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, test Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: HA branch (HDFS-1623) Attachments: hdfs-2622.txt Since HDFS-1975, we now increment the generation stamp for each block allocation. So the EXPECTED_TXID constant is wrong now in TestDFSUpgrade, since we do twice as many txns while creating files in the first phase of the test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2612) HA: handle refreshNameNodes in federated HA clusters
[
https://issues.apache.org/jira/browse/HDFS-2612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved HDFS-2612.
---
Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)
Hadoop Flags: Reviewed
Committed to branch, thanks for the review.
HA: handle refreshNameNodes in federated HA clusters
Key: HDFS-2612
URL: https://issues.apache.org/jira/browse/HDFS-2612
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node, ha
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: HA branch (HDFS-1623)
Attachments: hdfs-2612.txt, hdfs-2612.txt
For expediency in HDFS-1971 we've commented out the {{refreshNameNodes}}
function temporarily on branch. Need to fix that code to handle refresh with
HA.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2623) HA: Add test case for hot standby capability
[ https://issues.apache.org/jira/browse/HDFS-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-2623. --- Resolution: Fixed Fix Version/s: HA branch (HDFS-1623) Hadoop Flags: Reviewed Committed the failing test case so we can all share it while working towards getting the hot-standby in working shape. HA: Add test case for hot standby capability Key: HDFS-2623 URL: https://issues.apache.org/jira/browse/HDFS-2623 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, test Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: HA branch (HDFS-1623) Attachments: hdfs-2623.txt Putting up a fairly simple test case I wrote that verifies that the standby is kept hot -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2616) Change DatanodeProtocol#sendHeartbeat to return HeartbeatResponse
[ https://issues.apache.org/jira/browse/HDFS-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reopened HDFS-2616: --- Change DatanodeProtocol#sendHeartbeat to return HeartbeatResponse - Key: HDFS-2616 URL: https://issues.apache.org/jira/browse/HDFS-2616 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, name-node Affects Versions: HA branch (HDFS-1623) Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: HA branch (HDFS-1623) Attachments: HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, hdfs-2616-addendum.txt DatanodeProtocol#sendHeartbeat() returns DatanodeCommand[]. This jira proposes changing it to to return HeartbeatResponse that has DatanodeCommand[]. This allows adding other information that can be returned by the namenode to the datanode, instead of having to only return DatanodeCommand[]. For relevant discussion see HDFS-1972. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2616) Change DatanodeProtocol#sendHeartbeat to return HeartbeatResponse
[ https://issues.apache.org/jira/browse/HDFS-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-2616. --- Resolution: Fixed Committed hdfs-2616-addendum.txt as r1209315. Change DatanodeProtocol#sendHeartbeat to return HeartbeatResponse - Key: HDFS-2616 URL: https://issues.apache.org/jira/browse/HDFS-2616 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, name-node Affects Versions: HA branch (HDFS-1623) Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: HA branch (HDFS-1623) Attachments: HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, hdfs-2616-addendum.txt DatanodeProtocol#sendHeartbeat() returns DatanodeCommand[]. This jira proposes changing it to to return HeartbeatResponse that has DatanodeCommand[]. This allows adding other information that can be returned by the namenode to the datanode, instead of having to only return DatanodeCommand[]. For relevant discussion see HDFS-1972. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: another HDFS configuration for scribeH
Hi Hairong, What is the risk for this change? How much more testing do you think will be needed? -Eric From: Hairong Kuang hair...@fb.commailto:hair...@fb.com Date: Thu, 1 Dec 2011 20:22:10 -0800 To: Internal Use ehw...@fb.commailto:ehw...@fb.com Cc: Zheng Shao zs...@fb.commailto:zs...@fb.com, hdfs-dev@hadoop.apache.orgmailto:hdfs-dev@hadoop.apache.org hdfs-dev@hadoop.apache.orgmailto:hdfs-dev@hadoop.apache.org Subject: another HDFS configuration for scribeH Hi Eric, I was debugging a bizar data corruption case in the silver cluster today and realized that there is a very important configuration that scribeH cluster should set. Could you please set dfs.datanode.synconclose to be true in ScribeH for next week's push? This is will guarantee that block data get persisted to disk on close, so preventing data loss when datanodes get rebooted. Thanks, Hairong
another HDFS configuration for scribeH
Hi Eric, I was debugging a bizar data corruption case in the silver cluster today and realized that there is a very important configuration that scribeH cluster should set. Could you please set dfs.datanode.synconclose to be true in ScribeH for next week's push? This is will guarantee that block data get persisted to disk on close, so preventing data loss when datanodes get rebooted. Thanks, Hairong
RE: another HDFS configuration for scribeH
Hi Eric, Please check my recent experience with the dataloss issue. http://lucene.472066.n3.nabble.com/Blocks-are-getting-corrupted-under-very-high-load-tt3527403.html#a3532671 http://lucene.472066.n3.nabble.com/Update-RE-Blocks-are-getting-corrupted-under-very-high-load-tt3537420.html#none Performance degradtion is 9% with 10 Cleinet Threads , 8 node cluster, 24TB each machine. It is always safe to set this flag if your machine will get good amount of load. Regards, Uma From: Eric Hwang [ehw...@fb.com] Sent: Friday, December 02, 2011 9:56 AM To: Hairong Kuang Cc: Zheng Shao; hdfs-dev@hadoop.apache.org Subject: Re: another HDFS configuration for scribeH Hi Hairong, What is the risk for this change? How much more testing do you think will be needed? -Eric From: Hairong Kuang hair...@fb.commailto:hair...@fb.com Date: Thu, 1 Dec 2011 20:22:10 -0800 To: Internal Use ehw...@fb.commailto:ehw...@fb.com Cc: Zheng Shao zs...@fb.commailto:zs...@fb.com, hdfs-dev@hadoop.apache.orgmailto:hdfs-dev@hadoop.apache.org hdfs-dev@hadoop.apache.orgmailto:hdfs-dev@hadoop.apache.org Subject: another HDFS configuration for scribeH Hi Eric, I was debugging a bizar data corruption case in the silver cluster today and realized that there is a very important configuration that scribeH cluster should set. Could you please set dfs.datanode.synconclose to be true in ScribeH for next week's push? This is will guarantee that block data get persisted to disk on close, so preventing data loss when datanodes get rebooted. Thanks, Hairong
