[jira] [Created] (HDFS-7630) TestConnCache hardcode block size without considering native OS
sam liu created HDFS-7630: - Summary: TestConnCache hardcode block size without considering native OS Key: HDFS-7630 URL: https://issues.apache.org/jira/browse/HDFS-7630 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu Attachments: HDFS-7630.001.patch TestConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7629) TestDisableConnCache hardcode block size without considering native OS
sam liu created HDFS-7629: - Summary: TestDisableConnCache hardcode block size without considering native OS Key: HDFS-7629 URL: https://issues.apache.org/jira/browse/HDFS-7629 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestDisableConnCache hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7628) TestNameEditsConfigs hardcode block size without considering native OS
sam liu created HDFS-7628: - Summary: TestNameEditsConfigs hardcode block size without considering native OS Key: HDFS-7628 URL: https://issues.apache.org/jira/browse/HDFS-7628 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestNameEditsConfigs hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7627) TestCacheDirectives hardcode block size without considering native OS
sam liu created HDFS-7627: - Summary: TestCacheDirectives hardcode block size without considering native OS Key: HDFS-7627 URL: https://issues.apache.org/jira/browse/HDFS-7627 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestCacheDirectives hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7626) TestPipelinesFailover hardcode block size without considering native OS
sam liu created HDFS-7626: - Summary: TestPipelinesFailover hardcode block size without considering native OS Key: HDFS-7626 URL: https://issues.apache.org/jira/browse/HDFS-7626 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestPipelinesFailover hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7625) TestPersistBlocks hardcode block size without considering native OS
sam liu created HDFS-7625: - Summary: TestPersistBlocks hardcode block size without considering native OS Key: HDFS-7625 URL: https://issues.apache.org/jira/browse/HDFS-7625 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestPersistBlocks hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7624) TestFileAppendRestart hardcode block size without considering native OS
sam liu created HDFS-7624: - Summary: TestFileAppendRestart hardcode block size without considering native OS Key: HDFS-7624 URL: https://issues.apache.org/jira/browse/HDFS-7624 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: sam liu Assignee: sam liu TestFileAppendRestart hardcode block size with 'BLOCK_SIZE = 4096', however it's incorrect on some platforms. For example, on power platform, the correct value is 65536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7585) TestEnhancedByteBufferAccess hard code the block size
sam liu created HDFS-7585: - Summary: TestEnhancedByteBufferAccess hard code the block size Key: HDFS-7585 URL: https://issues.apache.org/jira/browse/HDFS-7585 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Assignee: sam liu Priority: Blocker The test TestEnhancedByteBufferAccess hard code the block size, and it fails with exceptions on power linux. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7114) Secondary NameNode failed to rollback from 2.4.1 to 2.2.0
sam liu created HDFS-7114: - Summary: Secondary NameNode failed to rollback from 2.4.1 to 2.2.0 Key: HDFS-7114 URL: https://issues.apache.org/jira/browse/HDFS-7114 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: sam liu Priority: Blocker Can upgrade from 2.2.0 to 2.4.1, but failed to rollback the secondary namenode with following issue. 2014-09-22 10:41:28,358 FATAL org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Failed to start secondary namenode org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /var/hadoop/tmp/hdfs/dfs/namesecondary. Reported: -56. Expecting = -47. at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1082) at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:890) at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:585) at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:921) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.recoverCreate(SecondaryNameNode.java:913) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:249) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.(SecondaryNameNode.java:199) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:652) 2014-09-22 10:41:28,360 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 2014-09-22 10:41:28,363 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG: -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7053) Failed to rollback hdfs version from 2.4.1 to 2.2.0
sam liu created HDFS-7053: - Summary: Failed to rollback hdfs version from 2.4.1 to 2.2.0 Key: HDFS-7053 URL: https://issues.apache.org/jira/browse/HDFS-7053 Project: Hadoop HDFS Issue Type: Bug Components: ha, namenode Affects Versions: 2.4.1 Reporter: sam liu Priority: Blocker I can successfully upgrade from 2.2.0 to 2.4.1 with QJM HA enabled and with downtime, but failed to rollback from 2.4.1 to 2.2.0. The error message: 2014-09-10 16:50:29,599 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join org.apache.hadoop.HadoopIllegalArgumentException: Invalid startup option. Cannot perform DFS upgrade with HA enabled. at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1207) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320) 2014-09-10 16:50:29,601 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7002) Failed to rolling upgrade hdfs from 2.2.0 to 2.4.1
sam liu created HDFS-7002: - Summary: Failed to rolling upgrade hdfs from 2.2.0 to 2.4.1 Key: HDFS-7002 URL: https://issues.apache.org/jira/browse/HDFS-7002 Project: Hadoop HDFS Issue Type: Bug Components: journal-node, namenode, qjm Affects Versions: 2.4.1, 2.2.0 Reporter: sam liu Priority: Blocker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-5046) Hang when add/remove a datanode into/from a 2 datanode cluster
sam liu created HDFS-5046: - Summary: Hang when add/remove a datanode into/from a 2 datanode cluster Key: HDFS-5046 URL: https://issues.apache.org/jira/browse/HDFS-5046 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 1.1.1 Environment: Red Hat Enterprise Linux Server release 5.3, 64 bit Reporter: sam liu 1. Install a Hadoop 1.1.1 cluster, with 2 datanodes: dn1 and dn2. And, in hdfs-site.xml, set the 'dfs.replication' to 2 2. Add node dn3 into the cluster as a new datanode, and did not change the 'dfs.replication' value in hdfs-site.xml and keep it as 2 note: step 2 passed 3. Decommission dn3 from the cluster Expected result: dn3 could be decommissioned successfully Actual result: a). decommission progress hangs and the status always be 'Waiting DataNode status: Decommissioned'. But, if I execute 'hadoop dfs -setrep -R 2 /', the decommission continues and will be completed finally. b). However, if the initial cluster includes >= 3 datanodes, this issue won't be encountered when add/remove another datanode. For example, if I setup a cluster with 3 datanodes, and then I can successfully add the 4th datanode into it, and then also can successfully remove the 4th datanode from the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4527) For shortening the time of TaskTracker heartbeat, decouple the statics collection operations
sam liu created HDFS-4527: - Summary: For shortening the time of TaskTracker heartbeat, decouple the statics collection operations Key: HDFS-4527 URL: https://issues.apache.org/jira/browse/HDFS-4527 Project: Hadoop HDFS Issue Type: Improvement Components: performance Affects Versions: 1.1.1 Reporter: sam liu In each heartbeat of TaskTracker, it will calculate some system statics, like the free disk space, available virtual/physical memory, cpu usage, etc. However, it's not necessary to calculate all the statics in every heartbeat, and this will consume many system resource and impace the performance of TaskTracker heartbeat. Furthermore, the characteristics of system properties(disk, memory, cpu) are different and it's better to collect their statics in different intervals. To reduce the latency of TaskTracker heartbeat, one solution is to decouple all the system statics collection operations from it, and issue separate threads to do the statics collection works when the TaskTracker starts. The threads could be three: the first one is to collect cpu related statics in a short interval; the second one is to collect memory related statics in a normal interval; the third one is to collect disk related statics in a long interval. And all the interval could be customized by the parameter "mapred.stats.collection.interval" in the mapred-site.xml. At last, the heartbeat could get values of system statics from the memory directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
