[jira] [Created] (HDFS-3375) Put client name in DataXceiver thread name for readBlock and keepalive

[Todd Lipcon (JIRA)]

Todd Lipcon created HDFS-3375:
-

 Summary: Put client name in DataXceiver thread name for readBlock 
and keepalive
 Key: HDFS-3375
 URL: https://issues.apache.org/jira/browse/HDFS-3375
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: data-node
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Trivial


Currently the datanode thread names include the client for write operations, 
but not for read. We should include the client name for read ops. Additionally, 
in the keepalive phase, we should include the client name that initiated the 
previous request as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3374) hdfs' TestDelegationToken fails intermittently with a race condition

[Owen O'Malley (JIRA)]

Owen O'Malley created HDFS-3374:
---

 Summary: hdfs' TestDelegationToken fails intermittently with a 
race condition
 Key: HDFS-3374
 URL: https://issues.apache.org/jira/browse/HDFS-3374
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Reporter: Owen O'Malley
Assignee: Owen O'Malley


The testcase is failing because the MiniDFSCluster is shutdown before the 
secret manager can change the key, which calls system.exit with no edit streams 
available.

{code}

[junit] 2012-05-04 15:03:51,521 WARN  common.Storage 
(FSImage.java:updateRemovedDirs(224)) - Removing storage dir 
/home/horton/src/hadoop/build/test/data/dfs/name1
[junit] 2012-05-04 15:03:51,522 FATAL namenode.FSNamesystem 
(FSEditLog.java:fatalExit(388)) - No edit streams are accessible
[junit] java.lang.Exception: No edit streams are accessible
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.fatalExit(FSEditLog.java:388)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.exitIfNoStreams(FSEditLog.java:407)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.removeEditsAndStorageDir(FSEditLog.java:432)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.removeEditsStreamsAndStorageDirs(FSEditLog.java:468)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:1028)
[junit] at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logUpdateMasterKey(FSNamesystem.java:5641)
[junit] at 
org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logUpdateMasterKey(DelegationTokenSecretManager.java:286)
[junit] at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:150)
[junit] at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.rollMasterKey(AbstractDelegationTokenSecretManager.java:174)
[junit] at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover.run(AbstractDelegationTokenSecretManager.java:385)
[junit] at java.lang.Thread.run(Thread.java:662)
[junit] Running org.apache.hadoop.hdfs.security.TestDelegationToken
[junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
[junit] Test org.apache.hadoop.hdfs.security.TestDelegationToken FAILED 
(crashed)
{code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3373) FileContext HDFS implementation can leak socket caches

[Todd Lipcon (JIRA)]

Todd Lipcon created HDFS-3373:
-

 Summary: FileContext HDFS implementation can leak socket caches
 Key: HDFS-3373
 URL: https://issues.apache.org/jira/browse/HDFS-3373
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs client
Affects Versions: 2.0.0
Reporter: Todd Lipcon


As noted by Nicholas in HDFS-3359, FileContext doesn't have a close() method, 
and thus never calls DFSClient.close(). This means that, until finalizers run, 
DFSClient will hold on to its SocketCache object and potentially have a lot of 
outstanding sockets/fds held on to.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3372) offlineEditsViewer should be able to read a binary edits file with recovery mode

[Colin Patrick McCabe (JIRA)]

Colin Patrick McCabe created HDFS-3372:
--

 Summary: offlineEditsViewer should be able to read a binary edits 
file with recovery mode
 Key: HDFS-3372
 URL: https://issues.apache.org/jira/browse/HDFS-3372
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor


It would be nice if oev (the offline edits viewer) had a switch that allowed us 
to read a binary edits file using recovery mode.  oev can be very useful when 
working with corrupt or messed up edit log files, and this would make it even 
more so.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-3345) Primary and secondary NameNode principals must be the same

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HDFS-3345.
--

Resolution: Duplicate

Closing this as a dupe. Please feel free to reopen it if you disagree, or if 
I've misinterpreted your report of the issue.

> Primary and secondary NameNode principals must be the same
> --
>
> Key: HDFS-3345
> URL: https://issues.apache.org/jira/browse/HDFS-3345
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Owen O'Malley
>
> The NameNode and SecondaryNameNode have two different configuration knobs 
> (dfs.namenode.kerberos.principal and 
> dfs.secondary.namenode.kerberos.principal), but the secondary namenode fails 
> authorization unless it is the same user.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3371) EditLogFileInputStream: be more careful about closing streams when we're done with them.

[Colin Patrick McCabe (JIRA)]

Colin Patrick McCabe created HDFS-3371:
--

 Summary: EditLogFileInputStream: be more careful about closing 
streams when we're done with them.
 Key: HDFS-3371
 URL: https://issues.apache.org/jira/browse/HDFS-3371
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor


EditLogFileInputStream#EditLogFileInputStream should be more careful about 
closing streams when there is an exception thrown.  Also, 
EditLogFileInputStream#close should close all of the streams we opened in the 
constructor, not just one of them (although the file-backed one is probably the 
most important).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3370) HDFS hardlink

[Hairong Kuang (JIRA)]

Hairong Kuang created HDFS-3370:
---

 Summary: HDFS hardlink
 Key: HDFS-3370
 URL: https://issues.apache.org/jira/browse/HDFS-3370
 Project: Hadoop HDFS
  Issue Type: New Feature
Reporter: Hairong Kuang
Assignee: Liyin Tang


We'd like to add a new feature to HDFS, hardlink, that allows harlinked files 
to share data without copying. Currently we will support hardlinking only 
closed files, but it could be extended to unclosed files as well.

Among many potential use cases of feature, the following two are primarily used 
in facebook:
1. This provides a lightweight way for applications like hbase to create a 
snapshot;
2. This also allows an application like Hive to move a table to a different 
directory without breaking current running hive queries.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HDFS-3362) Javadoc for FileStatus.getLen() is incorrect

[Oleksandr Alesinskyy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Alesinskyy reopened HDFS-3362:



That's true - but it is not fixed in 1.x branches, e.g. branch-1, which is 
marked as "sustaining branch for updates to Hadoop 1.0.0.". That means future 
minor version of 1.x.x likely would not have this fix.

 

> Javadoc for FileStatus.getLen() is incorrect
> 
>
> Key: HDFS-3362
> URL: https://issues.apache.org/jira/browse/HDFS-3362
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: documentation, hdfs client
>Affects Versions: 0.20.2, 1.0.2
>Reporter: Oleksandr Alesinskyy
>Priority: Minor
>
> {code}
>   /* 
>* @return the length of this file, in blocks
>*/
>   public long getLen() {
> return length;
>   }
> {code}
> in reality it return a length not in blocks but in bytes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3369) change variable names referring to inode in blockmanagement to more appropriate

[John George (JIRA)]

John George created HDFS-3369:
-

 Summary: change variable names referring to inode in 
blockmanagement to more appropriate
 Key: HDFS-3369
 URL: https://issues.apache.org/jira/browse/HDFS-3369
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: 2.0.0, 3.0.0
Reporter: John George
Assignee: John George
Priority: Minor


We should rename BlocksMap.getINode(..) and, in addition, the local variable 
names such as fileInode to match 'block collection'

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Hadoop-Hdfs-trunk #1034

[Apache Jenkins Server]

See 

Changes:

[tucu] HADOOP-8343. Allow configuration of authorization for JmxJsonServlet and 
MetricsServlet (tucu)

[tucu] MAPREDUCE-4205. retrofit all JVM shutdown hooks to use 
ShutdownHookManager (tucu)

[tucu] HADOOP-8355. SPNEGO filter throws/logs exception when authentication 
fails (tucu)

[tucu] HADOOP-8356. FileSystem service loading mechanism should print the 
FileSystem impl it is failing to load (tucu)

[szetszwo] HDFS-3350. In INode, add final to compareTo(..), equals(..) and 
hashCode(), and remove synchronized from updatePermissionStatus(..).

[todd] Amend previous commit of HADOOP-8350 (missed new SocketInputWrapper file)

[todd] HADOOP-8350. Improve NetUtils.getInputStream to return a stream which 
has a tunable timeout. Contributed by Todd Lipcon.

[todd] HDFS-3359. DFSClient.close should close cached sockets. Contributed by 
Todd Lipcon.

[umamahesh] HDFS-3332. NullPointerException in DN when directoryscanner is 
trying to report bad blocks. Contributed by  Amith D K.

[bobby] MAPREDUCE-4163. consistently set the bind address (Daryn Sharp via 
bobby)

[ddas] HADOOP-8346. Makes oid changes to make SPNEGO work. Was broken due to 
fixes introduced by the IBM JDK compatibility patch. Contributed by Devaraj Das.

--
[...truncated 11365 lines...]
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Building index for all the packages and classes...
Generating 

Hadoop-Hdfs-trunk - Build # 1034 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1034/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 11558 lines...]
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-hdfs-project 
---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:build-classpath (build-classpath) @ 
hadoop-hdfs-project ---
[INFO] Wrote classpath file 
'/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/target/classes/mrapp-generated-classpath'.
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.0:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  SUCCESS [5:29.163s]
[INFO] Apache Hadoop HttpFS .. SUCCESS [35.476s]
[INFO] Apache Hadoop HDFS BookKeeper Journal . SUCCESS [12.939s]
[INFO] Apache Hadoop HDFS Fuse ... SUCCESS [0.039s]
[INFO] Apache Hadoop HDFS Project  SUCCESS [0.034s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 6:18.467s
[INFO] Finished at: Fri May 04 11:40:57 UTC 2012
[INFO] Final Memory: 83M/746M
[INFO] 
+ /home/jenkins/tools/maven/latest/bin/mvn test 
-Dmaven.test.failure.ignore=true -Pclover 
-DcloverLicenseLocation=/home/jenkins/tools/clover/latest/lib/clover.license
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Publishing Javadoc
Recording fingerprints
Updating HADOOP-8356
Updating HDFS-3359
Updating MAPREDUCE-4163
Updating HADOOP-8350
Updating HDFS-3332
Updating MAPREDUCE-4205
Updating HADOOP-8355
Updating HADOOP-8343
Updating HDFS-3350
Updating HADOOP-8346
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Created] (HDFS-3368) Missing blocks due to bad DataNodes comming up and down.

[Konstantin Shvachko (JIRA)]

Konstantin Shvachko created HDFS-3368:
-

 Summary: Missing blocks due to bad DataNodes comming up and down.
 Key: HDFS-3368
 URL: https://issues.apache.org/jira/browse/HDFS-3368
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 1.0.0, 0.22.0, 2.0.0, 3.0.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko


All replicas of a block can be removed if bad DataNodes come up and down during 
cluter restart resulting in data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira