[jira] [Created] (HDFS-3346) add fuzz tester for FSImage serialization / deserializtion

[Colin Patrick McCabe (JIRA)]

Colin Patrick McCabe created HDFS-3346:
--

 Summary: add fuzz tester for FSImage serialization / deserializtion
 Key: HDFS-3346
 URL: https://issues.apache.org/jira/browse/HDFS-3346
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Colin Patrick McCabe
Priority: Minor


HDFS-3134 added a fuzz test for edit log serialization.  The goal of this test 
was to ensure that we got no runtime errors when deserializing incorrect edit 
log data, only IOExceptions.  We should have a similar test for FSImage 
deserialization.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3345) Primary and secondary Principals must be the same

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3345:
---

 Summary: Primary and secondary Principals must be the same
 Key: HDFS-3345
 URL: https://issues.apache.org/jira/browse/HDFS-3345
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Owen O'Malley


The NameNode and SecondaryNameNode have two different configuration knobs 
(dfs.namenode.kerberos.principal and 
dfs.secondary.namenode.kerberos.principal), but the secondary namenode fails 
authorization unless it is the same user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3344) TestProcessCorruptBlocks sometimes fails

[Tsz Wo (Nicholas), SZE (JIRA)]

Tsz Wo (Nicholas), SZE created HDFS-3344:


 Summary: TestProcessCorruptBlocks sometimes fails
 Key: HDFS-3344
 URL: https://issues.apache.org/jira/browse/HDFS-3344
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Tsz Wo (Nicholas), SZE


It failed in [build 
#2356|https://builds.apache.org/job/PreCommit-HDFS-Build/2356//testReport/org.apache.hadoop.hdfs.server.namenode/TestProcessCorruptBlocks/testWithReplicationFactorAsOne/]:
{noformat}
java.lang.AssertionError: expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at 
org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks.testWithReplicationFactorAsOne(TestProcessCorruptBlocks.java:192)
...
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3343) Improve metrics for DN read latency

[Todd Lipcon (JIRA)]

Todd Lipcon created HDFS-3343:
-

 Summary: Improve metrics for DN read latency
 Key: HDFS-3343
 URL: https://issues.apache.org/jira/browse/HDFS-3343
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: data-node
Reporter: Todd Lipcon


Similar to HDFS-3170 on the write side, we should improve the metrics that are 
generated on the DN for read latency. We should have separate metrics for the 
time spent in {{transferTo}} vs {{waitWritable}} so that it's easy to 
distinguish slow local disks from slow readers on the other end of the socket.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3342) SocketTimeoutException in BlockSender.sendChunks could have a better error message

[Todd Lipcon (JIRA)]

Todd Lipcon created HDFS-3342:
-

 Summary: SocketTimeoutException in BlockSender.sendChunks could 
have a better error message
 Key: HDFS-3342
 URL: https://issues.apache.org/jira/browse/HDFS-3342
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: data-node
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Priority: Minor


Currently, if a client connects to a DN and begins to read a block, but then 
stops calling read() for a long period of time, the DN will log a 
SocketTimeoutException "48 millis timeout while waiting for channel to be 
ready for write." This is because there is no "keepalive" functionality of any 
kind. At a minimum, we should improve this error message to be an INFO level 
log which just says that the client likely stopped reading, so disconnecting it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3341) Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0

[Todd Lipcon (JIRA)]

Todd Lipcon created HDFS-3341:
-

 Summary: Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 
2.0.0
 Key: HDFS-3341
 URL: https://issues.apache.org/jira/browse/HDFS-3341
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 2.0.0, 3.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
 Attachments: hdfs-3341.txt

After we commit HADOOP-8340, tests will fail because the minimum version is 
configured to be 2.0.0/3.0.0 instead of the corresponding SNAPSHOT builds. When 
we commit HADOOP-8340, we should update the minimums at the same time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-3340) fix FSEditLog#verifyEndOfLog

[Colin Patrick McCabe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe resolved HDFS-3340.


  Resolution: Invalid
Release Note: my mistake, this didn't go in in the form I thought it did

> fix FSEditLog#verifyEndOfLog
> 
>
> Key: HDFS-3340
> URL: https://issues.apache.org/jira/browse/HDFS-3340
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>
> FSEditLog#verifyEndOfLog may incorrectly report corruption in some cases.  It 
> should be fixed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3340) fix FSEditLog#verifyEndOfLog

[Colin Patrick McCabe (JIRA)]

Colin Patrick McCabe created HDFS-3340:
--

 Summary: fix FSEditLog#verifyEndOfLog
 Key: HDFS-3340
 URL: https://issues.apache.org/jira/browse/HDFS-3340
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe


FSEditLog#verifyEndOfLog may incorrectly report corruption in some cases.  It 
should be fixed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3338) change INode to package private

[John George (JIRA)]

John George created HDFS-3338:
-

 Summary: change INode to package private
 Key: HDFS-3338
 URL: https://issues.apache.org/jira/browse/HDFS-3338
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: 2.0.0, 3.0.0
Reporter: John George
Assignee: John George
Priority: Minor


Only a few place in blockmanagement is using INode and all of them can be 
replaced by INodeFile.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3339) change INode to package private

[John George (JIRA)]

John George created HDFS-3339:
-

 Summary: change INode to package private
 Key: HDFS-3339
 URL: https://issues.apache.org/jira/browse/HDFS-3339
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: 2.0.0, 3.0.0
Reporter: John George
Assignee: John George
Priority: Minor


Only a few place in blockmanagement is using INode and all of them can be 
replaced by INodeFile.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3337) HDFS command fails with exception following merge of HADOOP-8325

[Randy Clayton (JIRA)]

Randy Clayton created HDFS-3337:
---

 Summary: HDFS command fails with exception following merge of 
HADOOP-8325
 Key: HDFS-3337
 URL: https://issues.apache.org/jira/browse/HDFS-3337
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 2.0.0
 Environment: QE tests on version 2.0.1205010603
Reporter: Randy Clayton


We are seeing most hdfs commands in our nightly acceptance tests fail with an 
exception as shown below. This started with a few hours of the merge of 
HADOOP-8325 on 4/30/2012

hdfs --config conf/hadoop/ dfs -ls dirname
ls: `dirname': No such file or directory
12/05/01 16:57:52 WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' 
failed, java.lang.IllegalStateException: Shutdown in progress, cannot remove a 
shutdownHook
java.lang.IllegalStateException: Shutdown in progress, cannot remove a 
shutdownHook
at 
org.apache.hadoop.util.ShutdownHookManager.removeShutdownHook(ShutdownHookManager.java:166)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2202)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2231)
at 
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2251)
at 
org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3336) hdfs launcher script will be better off not special casing namenode command with regards to hadoop.security.logger

[Roman Shaposhnik (JIRA)]

Roman Shaposhnik created HDFS-3336:
--

 Summary: hdfs launcher script will be better off not special 
casing namenode command with regards to hadoop.security.logger
 Key: HDFS-3336
 URL: https://issues.apache.org/jira/browse/HDFS-3336
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: scripts
Affects Versions: 0.23.1
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Priority: Minor
 Fix For: 2.0.0


All of the launcher scripts (except hdfs when called with a namenode argument) 
are defaulting to INFO,NullAppender setting for hadoop.security.logger. The 
only codepath where the default is INFO,RFAS is via hadoop-daemon.sh. This is 
exactly the right thing to do since only daemons are guaranteed to be executed 
under the proper user account (the one that has write/create access to the 
locations specified in ${hadoop.log.dir}/*).

It would be nice for 'hdfs namenode' to follow the suit and not produce 
spurious warnings about not being able to write to log locations when executed 
from under regular accounts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Hadoop-Hdfs-trunk - Build # 1031 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1031/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 11129 lines...]

main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ 
hadoop-hdfs-project ---
[INFO] Wrote classpath file 
'/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/target/classes/mrapp-generated-classpath'.
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.0:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  SUCCESS [5:27.429s]
[INFO] Apache Hadoop HttpFS .. SUCCESS [35.500s]
[INFO] Apache Hadoop HDFS BookKeeper Journal . SUCCESS [13.012s]
[INFO] Apache Hadoop HDFS Fuse ... SUCCESS [0.039s]
[INFO] Apache Hadoop HDFS Project  SUCCESS [0.033s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 6:16.825s
[INFO] Finished at: Tue May 01 11:40:46 UTC 2012
[INFO] Final Memory: 88M/742M
[INFO] 
+ /home/jenkins/tools/maven/latest/bin/mvn test 
-Dmaven.test.failure.ignore=true -Pclover 
-DcloverLicenseLocation=/home/jenkins/tools/clover/latest/lib/clover.license
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Publishing Javadoc
Recording fingerprints
Updating HDFS-3293
Updating HADOOP-8305
Updating MAPREDUCE-4209
Updating HADOOP-8308
Updating MAPREDUCE-4206
Updating MAPREDUCE-3883
Updating HADOOP-8312
Updating HADOOP-8330
Updating HADOOP-8325
Updating HADOOP-8335
Updating HDFS-3275
Updating HADOOP-8334
Updating HDFS-3286
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
All tests passed

Build failed in Jenkins: Hadoop-Hdfs-trunk #1031

[Apache Jenkins Server]

See 

Changes:

[umamahesh] HDFS-3286. When the threshold value for balancer is zero, 
unexpected output is displayed. Contributed by Ashish Singhi.

[umamahesh] HDFS-3275. Skip format for non-file based directories. Contributed 
by Amith D K.

[tomwhite] HADOOP-8308. Support cross-project Jenkins builds.

[szetszwo] HDFS-3293. Add toString(), equals(..) and hashCode() to JournalInfo. 
 Contributed by Hari Mankude

[bobby] HADOOP-8335. Improve Configuration's address handling (Daryn Sharp via 
bobby)

[bobby] HADOOP-8312. testpatch.sh should provide a simpler way to see which 
warnings changed (bobby)

[szetszwo] HADOOP-8330. Update TestSequenceFile.testCreateUsesFsArg() for 
HADOOP-8305.  Contributed by John George

[tucu] HADOOP-8325. Add a ShutdownHookManager to be used by different 
components instead of the JVM shutdownhook (tucu)

[bobby] HADOOP-8334. HttpServer sometimes returns incorrect port (Daryn Sharp 
via bobby)

[tgraves] MAPREDUCE-4206. Sorting by Last Health-Update on the RM nodes page 
sorts does not work correctly (Jonathon Eagles via tgraves)

[bobby] MAPREDUCE-4209. junit dependency in hadoop-mapreduce-client is missing 
scope test (Radim Kolar via bobby)

[tgraves] MAPREDUCE-3883. Document yarn.nodemanager.delete.debug-delay-sec 
configuration property (Eugene Koontz via tgraves)

--
[...truncated 10936 lines...]
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Building index for all the packages and classes...
Generating 

Generating 

Generating 

Building index for all classes...
Generating 

Generating 

Jenkins build is back to stable : Hadoop-Hdfs-0.23-Build #244

[Apache Jenkins Server]

See