[jira] [Created] (HDFS-6903) Crc32 checksum errors in Big-Endian Architecture
Ayappan created HDFS-6903: - Summary: Crc32 checksum errors in Big-Endian Architecture Key: HDFS-6903 URL: https://issues.apache.org/jira/browse/HDFS-6903 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.4.1, 3.0.0 Environment: PowerPC RHEL 7 & 6.5 ( ppc64 - Big-Endian ) Reporter: Ayappan Fix For: 2.4.1 Native Crc32 checksum calculation is not handled in Big-Endian Architecture.In this case, the platform is ppc64. Due to this several testcases in HDFS module fails. Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) Time elapsed: 7.141 sec <<< ERROR! java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native Method) at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) at org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) testSwitchAlgorithms(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) Time elapsed: 1.394 sec <<< ERROR! java.io.IOException: p=/testSwitchAlgorithms, length=3000, i=0 at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native Method) at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) at org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testSwitchAlgorithms(TestAppendDifferentChecksum.java:94) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6898: Attachment: HDFS-6898.01.patch > DN must reserve space for a full block when an RBW block is created > --- > > Key: HDFS-6898 > URL: https://issues.apache.org/jira/browse/HDFS-6898 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Gopal V >Assignee: Arpit Agarwal > Attachments: HDFS-6898.01.patch > > > DN will successfully create two RBW blocks on the same volume even if the > free space is sufficient for just one full block. > One or both block writers may subsequently get a DiskOutOfSpace exception. > This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6903) Crc32 checksum errors in Big-Endian Architecture
[ https://issues.apache.org/jira/browse/HDFS-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayappan updated HDFS-6903: -- Attachment: HDFS-6903.patch > Crc32 checksum errors in Big-Endian Architecture > > > Key: HDFS-6903 > URL: https://issues.apache.org/jira/browse/HDFS-6903 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.4.1 > Environment: PowerPC RHEL 7 & 6.5 ( ppc64 - Big-Endian ) >Reporter: Ayappan > Fix For: 2.4.1 > > Attachments: HDFS-6903.patch > > > Native Crc32 checksum calculation is not handled in Big-Endian > Architecture.In this case, the platform is ppc64. Due to this several > testcases in HDFS module fails. > Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum > Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec > <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum > testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 7.141 sec <<< ERROR! > java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) > testSwitchAlgorithms(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 1.394 sec <<< ERROR! > java.io.IOException: p=/testSwitchAlgorithms, length=3000, i=0 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testSwitchAlgorithms(TestAppendDifferentChecksum.java:94) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6903) Crc32 checksum errors in Big-Endian Architecture
[ https://issues.apache.org/jira/browse/HDFS-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayappan updated HDFS-6903: -- Description: Native Crc32 checksum calculation is not handled in Big-Endian Architecture.In this case, the platform is ppc64. Due to this several testcases in HDFS module fails. The attached Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) Time elapsed: 7.141 sec <<< ERROR! java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native Method) at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) at org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) testSwitchAlgorithms(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) Time elapsed: 1.394 sec <<< ERROR! java.io.IOException: p=/testSwitchAlgorithms, length=3000, i=0 at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native Method) at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) at org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testSwitchAlgorithms(TestAppendDifferentChecksum.java:94) was: Native Crc32 checksum calculation is not handled in Big-Endian Architecture.In this case, the platform is ppc64. Due to this several testcases in HDFS module fails. Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) Time elapsed: 7.141 sec <<< ERROR! java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native Method) at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) at org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) testSwitchAlg
[jira] [Updated] (HDFS-6903) Crc32 checksum errors in Big-Endian Architecture
[ https://issues.apache.org/jira/browse/HDFS-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayappan updated HDFS-6903: -- Description: Native Crc32 checksum calculation is not handled in Big-Endian Architecture.In this case, the platform is ppc64. Due to this several testcases in HDFS module fails. Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) Time elapsed: 7.141 sec <<< ERROR! java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native Method) at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) at org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) testSwitchAlgorithms(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) Time elapsed: 1.394 sec <<< ERROR! java.io.IOException: p=/testSwitchAlgorithms, length=3000, i=0 at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native Method) at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) at org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testSwitchAlgorithms(TestAppendDifferentChecksum.java:94) was: Native Crc32 checksum calculation is not handled in Big-Endian Architecture.In this case, the platform is ppc64. Due to this several testcases in HDFS module fails. The attached Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) Time elapsed: 7.141 sec <<< ERROR! java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native Method) at org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) at org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) at org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) at org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) testSwitchAlg
[jira] [Commented] (HDFS-6827) Both NameNodes stuck in STANDBY state due to HealthMonitor not aware of the target's status changing sometimes
[
https://issues.apache.org/jira/browse/HDFS-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105261#comment-14105261
]
Zesheng Wu commented on HDFS-6827:
--
Ping [~vinayrpet]..
> Both NameNodes stuck in STANDBY state due to HealthMonitor not aware of the
> target's status changing sometimes
> --
>
> Key: HDFS-6827
> URL: https://issues.apache.org/jira/browse/HDFS-6827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.4.1
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
>Priority: Critical
> Attachments: HDFS-6827.1.patch
>
>
> In our production cluster, we encounter a scenario like this: ANN crashed due
> to write journal timeout, and was restarted by the watchdog automatically,
> but after restarting both of the NNs are standby.
> Following is the logs of the scenario:
> # NN1 is down due to write journal timeout:
> {color:red}2014-08-03,23:02:02,219{color} INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG
> # ZKFC1 detected "connection reset by peer"
> {color:red}2014-08-03,23:02:02,560{color} ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:xx@xx.HADOOP (auth:KERBEROS) cause:java.io.IOException:
> {color:red}Connection reset by peer{color}
> # NN1 wat restarted successfully by the watchdog:
> 2014-08-03,23:02:07,884 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> Web-server up at: xx:13201
> 2014-08-03,23:02:07,884 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> {color:red}2014-08-03,23:02:07,884{color} INFO org.apache.hadoop.ipc.Server:
> IPC Server listener on 13200: starting
> 2014-08-03,23:02:08,742 INFO org.apache.hadoop.ipc.Server: RPC server clean
> thread started!
> 2014-08-03,23:02:08,743 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> Registered DFSClientInformation MBean
> 2014-08-03,23:02:08,744 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> NameNode up at: xx/xx:13200
> 2014-08-03,23:02:08,744 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for standby state
> # ZKFC1 retried the connection and considered NN1 was healthy
> {color:red}2014-08-03,23:02:08,292{color} INFO org.apache.hadoop.ipc.Client:
> Retrying connect to server: xx/xx:13200. Already tried 0 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1
> SECONDS)
> # ZKFC1 still considered NN1 as a healthy Active NN, and didn't trigger the
> failover, as a result, both NNs were standby.
> The root cause of this bug is that NN is restarted too quickly and ZKFC
> health monitor doesn't realize that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
Varun Vasudev created HDFS-6904:
---
Summary: YARN unable to renew delegation token fetched via webhdfs
due to incorrect service port
Key: HDFS-6904
URL: https://issues.apache.org/jira/browse/HDFS-6904
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Reporter: Varun Vasudev
YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
The scenario is as follows -
1. User creates a delegation token using the WebHDFS REST API
2. User passes this token to YARN as part of app submission
3. When YARN tries to renew this delegation token, it fails because the token
service is pointing to the RPC port but the token kind is WebHDFS.
The exception is
{noformat}
2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
(DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation, Service:
NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
op=RENEWDELEGATIONTOKEN, message=null
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
at
org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
at org.apache.hadoop.security.token.Token.renew(Token.java:377)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
... 6 more
Caused by: java.io.IOException: The error stream is null.
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
... 24 more
2014-08-19 03:12:54,735 DEBUG event.AsyncDispatcher
(AsyncDispatcher.java:dispatch(164)) - Dispatching the event
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRejectedEvent.EventType:
APP_REJECTED
{noformat}
The issue is that the Namenode generates a delegation token of kind WebHDFS but
doesn't change the service port. When YARN tries to renew the delegation token,
it ends up trying to contact WebHDFS on the RPC port.
>From NamenodeWebHdfsMethods.java
{noformat}
case GETDELEGATIONTOKEN:
{
if (delegation.getValue() != null) {
throw new IllegalArgumentException(delegation.getName()
+ " parameter is not null.");
}
final T
[jira] [Commented] (HDFS-6901) Remove unnecessary CryptoCodec and KeyProvider.Options definition in FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105302#comment-14105302
]
Hadoop QA commented on HDFS-6901:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663326/HDFS-6901.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 3
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7700//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7700//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7700//console
This message is automatically generated.
> Remove unnecessary CryptoCodec and KeyProvider.Options definition in
> FSNamesystem
> -
>
> Key: HDFS-6901
> URL: https://issues.apache.org/jira/browse/HDFS-6901
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
>Priority: Minor
> Attachments: HDFS-6901.001.patch
>
>
> CryptoCodec and KeyProvider.Options are not necessary in FSN.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6903) Crc32 checksum errors in Big-Endian Architecture
[ https://issues.apache.org/jira/browse/HDFS-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayappan updated HDFS-6903: -- Attachment: (was: HDFS-6903.patch) > Crc32 checksum errors in Big-Endian Architecture > > > Key: HDFS-6903 > URL: https://issues.apache.org/jira/browse/HDFS-6903 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.4.1 > Environment: PowerPC RHEL 7 & 6.5 ( ppc64 - Big-Endian ) >Reporter: Ayappan > Fix For: 2.4.1 > > > Native Crc32 checksum calculation is not handled in Big-Endian > Architecture.In this case, the platform is ppc64. Due to this several > testcases in HDFS module fails. > Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum > Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec > <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum > testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 7.141 sec <<< ERROR! > java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) > testSwitchAlgorithms(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 1.394 sec <<< ERROR! > java.io.IOException: p=/testSwitchAlgorithms, length=3000, i=0 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testSwitchAlgorithms(TestAppendDifferentChecksum.java:94) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6899) Allow changing MiniDFSCluster volumes per DN and capacity per volume
[
https://issues.apache.org/jira/browse/HDFS-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105307#comment-14105307
]
Hadoop QA commented on HDFS-6899:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663313/HDFS-6899.01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 3
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7701//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7701//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7701//console
This message is automatically generated.
> Allow changing MiniDFSCluster volumes per DN and capacity per volume
>
>
> Key: HDFS-6899
> URL: https://issues.apache.org/jira/browse/HDFS-6899
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, test
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-6899.01.patch
>
>
> MiniDFSCluster hardcodes the number of directories per volume to two. Propose
> removing the hard-coded restriction.
> It would be useful to limit the capacity of individual storage directories
> for testing purposes. There is already a way to do so for SimulatedFSDataset,
> we can add one when using real volumes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated HDFS-6904:
Description:
YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
The scenario is as follows -
1. User creates a delegation token using the WebHDFS REST API
2. User passes this token to YARN as part of app submission(via the YARN REST
API)
3. When YARN tries to renew this delegation token, it fails because the token
service is pointing to the RPC port but the token kind is WebHDFS.
The exception is
{noformat}
2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
(DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation, Service:
NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
op=RENEWDELEGATIONTOKEN, message=null
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
at
org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
at org.apache.hadoop.security.token.Token.renew(Token.java:377)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
... 6 more
Caused by: java.io.IOException: The error stream is null.
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
... 24 more
2014-08-19 03:12:54,735 DEBUG event.AsyncDispatcher
(AsyncDispatcher.java:dispatch(164)) - Dispatching the event
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRejectedEvent.EventType:
APP_REJECTED
{noformat}
The issue is that the Namenode generates a delegation token of kind WebHDFS but
doesn't change the service port. When YARN tries to renew the delegation token,
it ends up trying to contact WebHDFS on the RPC port.
>From NamenodeWebHdfsMethods.java
{noformat}
case GETDELEGATIONTOKEN:
{
if (delegation.getValue() != null) {
throw new IllegalArgumentException(delegation.getName()
+ " parameter is not null.");
}
final Token token = generateDelegationToken(
namenode, ugi, renewer.getValue());
final String js = JsonUtil.toJsonString(token);
return Response.ok(js).type(Med
[jira] [Commented] (HDFS-6827) Both NameNodes stuck in STANDBY state due to HealthMonitor not aware of the target's status changing sometimes
[
https://issues.apache.org/jira/browse/HDFS-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105330#comment-14105330
]
Vinayakumar B commented on HDFS-6827:
-
Hi [~wuzesheng],
Please check the {{ZKFailOverController#verifyChangedServiceState(..)}} this
will be called for every health check callback, which will happen every 1 sec
by default.
In your case,
1. Service is HEALTHY, even though NN was restarted within this 1 sec interval.
2. But, after restart of NN, first healthcheck callback will identify state
change of ANN in {{ZKFailOverController#verifyChangedServiceState(..)}} and
ZKFC will quit the election marking {{quitElectionOnBadState}} to true. Now
another STANDBY ZKFC will have a chance to become ACTIVE.
3. Next callback from healthcheck will call {{rechedElectability()}} which
will in-turn make ZKFC to join the election back if the service is still
HEALTHY. Meanwhile other ZKFC would have won the leader election and became
ACTIVE.
So after HADOOP-10251, I feel your problem also will be solved.
Have you tried same scenario in latest trunk code?
> Both NameNodes stuck in STANDBY state due to HealthMonitor not aware of the
> target's status changing sometimes
> --
>
> Key: HDFS-6827
> URL: https://issues.apache.org/jira/browse/HDFS-6827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.4.1
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
>Priority: Critical
> Attachments: HDFS-6827.1.patch
>
>
> In our production cluster, we encounter a scenario like this: ANN crashed due
> to write journal timeout, and was restarted by the watchdog automatically,
> but after restarting both of the NNs are standby.
> Following is the logs of the scenario:
> # NN1 is down due to write journal timeout:
> {color:red}2014-08-03,23:02:02,219{color} INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG
> # ZKFC1 detected "connection reset by peer"
> {color:red}2014-08-03,23:02:02,560{color} ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:xx@xx.HADOOP (auth:KERBEROS) cause:java.io.IOException:
> {color:red}Connection reset by peer{color}
> # NN1 wat restarted successfully by the watchdog:
> 2014-08-03,23:02:07,884 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> Web-server up at: xx:13201
> 2014-08-03,23:02:07,884 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> {color:red}2014-08-03,23:02:07,884{color} INFO org.apache.hadoop.ipc.Server:
> IPC Server listener on 13200: starting
> 2014-08-03,23:02:08,742 INFO org.apache.hadoop.ipc.Server: RPC server clean
> thread started!
> 2014-08-03,23:02:08,743 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> Registered DFSClientInformation MBean
> 2014-08-03,23:02:08,744 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> NameNode up at: xx/xx:13200
> 2014-08-03,23:02:08,744 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for standby state
> # ZKFC1 retried the connection and considered NN1 was healthy
> {color:red}2014-08-03,23:02:08,292{color} INFO org.apache.hadoop.ipc.Client:
> Retrying connect to server: xx/xx:13200. Already tried 0 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1
> SECONDS)
> # ZKFC1 still considered NN1 as a healthy Active NN, and didn't trigger the
> failover, as a result, both NNs were standby.
> The root cause of this bug is that NN is restarted too quickly and ZKFC
> health monitor doesn't realize that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6827) Both NameNodes stuck in STANDBY state due to HealthMonitor not aware of the target's status changing sometimes
[
https://issues.apache.org/jira/browse/HDFS-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105331#comment-14105331
]
Vinayakumar B commented on HDFS-6827:
-
Update: In point #2, quitting the election will happen on second healthcheck
callback. two continuous callbacks will be checked before quitting election to
make sure that state change differences are not coming due to parellel
transition. And on third callback {{rechedElectability(..)}} will be called.
> Both NameNodes stuck in STANDBY state due to HealthMonitor not aware of the
> target's status changing sometimes
> --
>
> Key: HDFS-6827
> URL: https://issues.apache.org/jira/browse/HDFS-6827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 2.4.1
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
>Priority: Critical
> Attachments: HDFS-6827.1.patch
>
>
> In our production cluster, we encounter a scenario like this: ANN crashed due
> to write journal timeout, and was restarted by the watchdog automatically,
> but after restarting both of the NNs are standby.
> Following is the logs of the scenario:
> # NN1 is down due to write journal timeout:
> {color:red}2014-08-03,23:02:02,219{color} INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG
> # ZKFC1 detected "connection reset by peer"
> {color:red}2014-08-03,23:02:02,560{color} ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:xx@xx.HADOOP (auth:KERBEROS) cause:java.io.IOException:
> {color:red}Connection reset by peer{color}
> # NN1 wat restarted successfully by the watchdog:
> 2014-08-03,23:02:07,884 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> Web-server up at: xx:13201
> 2014-08-03,23:02:07,884 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> {color:red}2014-08-03,23:02:07,884{color} INFO org.apache.hadoop.ipc.Server:
> IPC Server listener on 13200: starting
> 2014-08-03,23:02:08,742 INFO org.apache.hadoop.ipc.Server: RPC server clean
> thread started!
> 2014-08-03,23:02:08,743 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> Registered DFSClientInformation MBean
> 2014-08-03,23:02:08,744 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
> NameNode up at: xx/xx:13200
> 2014-08-03,23:02:08,744 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for standby state
> # ZKFC1 retried the connection and considered NN1 was healthy
> {color:red}2014-08-03,23:02:08,292{color} INFO org.apache.hadoop.ipc.Client:
> Retrying connect to server: xx/xx:13200. Already tried 0 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=1, sleepTime=1
> SECONDS)
> # ZKFC1 still considered NN1 as a healthy Active NN, and didn't trigger the
> failover, as a result, both NNs were standby.
> The root cause of this bug is that NN is restarted too quickly and ZKFC
> health monitor doesn't realize that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6886) Use single editlog record for creating file + overwrite.
[ https://issues.apache.org/jira/browse/HDFS-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-6886: - Attachment: editsStored HDFS-6886.001.patch *1*, The patch makes one single editlog record for creating file + overwrite. *2*, Blocks of old file are removed after logSync, and we have discussed this in HDFS-6871, it's the same handle as delete file. > Use single editlog record for creating file + overwrite. > > > Key: HDFS-6886 > URL: https://issues.apache.org/jira/browse/HDFS-6886 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Yi Liu >Assignee: Yi Liu >Priority: Critical > Attachments: HDFS-6886.001.patch, editsStored > > > As discussed in HDFS-6871, as [~jingzhao] and [~cmccabe]'s suggestion, we > could do further improvement to use one editlog record for creating file + > overwrite in this JIRA. We could record the overwrite flag in editlog for > creating file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6886) Use single editlog record for creating file + overwrite.
[ https://issues.apache.org/jira/browse/HDFS-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-6886: - Status: Patch Available (was: In Progress) > Use single editlog record for creating file + overwrite. > > > Key: HDFS-6886 > URL: https://issues.apache.org/jira/browse/HDFS-6886 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Yi Liu >Assignee: Yi Liu >Priority: Critical > Attachments: HDFS-6886.001.patch, editsStored > > > As discussed in HDFS-6871, as [~jingzhao] and [~cmccabe]'s suggestion, we > could do further improvement to use one editlog record for creating file + > overwrite in this JIRA. We could record the overwrite flag in editlog for > creating file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6886) Use single editlog record for creating file + overwrite.
[ https://issues.apache.org/jira/browse/HDFS-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-6886: - Attachment: HDFS-6886.001.patch > Use single editlog record for creating file + overwrite. > > > Key: HDFS-6886 > URL: https://issues.apache.org/jira/browse/HDFS-6886 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Yi Liu >Assignee: Yi Liu >Priority: Critical > Attachments: HDFS-6886.001.patch, editsStored > > > As discussed in HDFS-6871, as [~jingzhao] and [~cmccabe]'s suggestion, we > could do further improvement to use one editlog record for creating file + > overwrite in this JIRA. We could record the overwrite flag in editlog for > creating file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6886) Use single editlog record for creating file + overwrite.
[ https://issues.apache.org/jira/browse/HDFS-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-6886: - Attachment: (was: HDFS-6886.001.patch) > Use single editlog record for creating file + overwrite. > > > Key: HDFS-6886 > URL: https://issues.apache.org/jira/browse/HDFS-6886 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Yi Liu >Assignee: Yi Liu >Priority: Critical > Attachments: HDFS-6886.001.patch, editsStored > > > As discussed in HDFS-6871, as [~jingzhao] and [~cmccabe]'s suggestion, we > could do further improvement to use one editlog record for creating file + > overwrite in this JIRA. We could record the overwrite flag in editlog for > creating file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6903) Crc32 checksum errors in Big-Endian Architecture
[ https://issues.apache.org/jira/browse/HDFS-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-6903: --- Priority: Blocker (was: Major) > Crc32 checksum errors in Big-Endian Architecture > > > Key: HDFS-6903 > URL: https://issues.apache.org/jira/browse/HDFS-6903 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.4.1 > Environment: PowerPC RHEL 7 & 6.5 ( ppc64 - Big-Endian ) >Reporter: Ayappan >Priority: Blocker > Fix For: 2.4.1 > > > Native Crc32 checksum calculation is not handled in Big-Endian > Architecture.In this case, the platform is ppc64. Due to this several > testcases in HDFS module fails. > Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum > Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec > <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum > testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 7.141 sec <<< ERROR! > java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) > testSwitchAlgorithms(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 1.394 sec <<< ERROR! > java.io.IOException: p=/testSwitchAlgorithms, length=3000, i=0 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testSwitchAlgorithms(TestAppendDifferentChecksum.java:94) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6903) Crc32 checksum errors in Big-Endian Architecture
[ https://issues.apache.org/jira/browse/HDFS-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-6903: --- Target Version/s: (was: 2.5.1) > Crc32 checksum errors in Big-Endian Architecture > > > Key: HDFS-6903 > URL: https://issues.apache.org/jira/browse/HDFS-6903 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.4.1, 2.6.0 > Environment: PowerPC RHEL 7 & 6.5 ( ppc64 - Big-Endian ) >Reporter: Ayappan >Priority: Blocker > > Native Crc32 checksum calculation is not handled in Big-Endian > Architecture.In this case, the platform is ppc64. Due to this several > testcases in HDFS module fails. > Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum > Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec > <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum > testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 7.141 sec <<< ERROR! > java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) > testSwitchAlgorithms(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 1.394 sec <<< ERROR! > java.io.IOException: p=/testSwitchAlgorithms, length=3000, i=0 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testSwitchAlgorithms(TestAppendDifferentChecksum.java:94) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6903) Crc32 checksum errors in Big-Endian Architecture
[ https://issues.apache.org/jira/browse/HDFS-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-6903: --- Fix Version/s: (was: 2.4.1) > Crc32 checksum errors in Big-Endian Architecture > > > Key: HDFS-6903 > URL: https://issues.apache.org/jira/browse/HDFS-6903 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.4.1, 2.6.0 > Environment: PowerPC RHEL 7 & 6.5 ( ppc64 - Big-Endian ) >Reporter: Ayappan >Priority: Blocker > > Native Crc32 checksum calculation is not handled in Big-Endian > Architecture.In this case, the platform is ppc64. Due to this several > testcases in HDFS module fails. > Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum > Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec > <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum > testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 7.141 sec <<< ERROR! > java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) > testSwitchAlgorithms(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 1.394 sec <<< ERROR! > java.io.IOException: p=/testSwitchAlgorithms, length=3000, i=0 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testSwitchAlgorithms(TestAppendDifferentChecksum.java:94) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6903) Crc32 checksum errors in Big-Endian Architecture
[ https://issues.apache.org/jira/browse/HDFS-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-6903: --- Affects Version/s: 2.6.0 > Crc32 checksum errors in Big-Endian Architecture > > > Key: HDFS-6903 > URL: https://issues.apache.org/jira/browse/HDFS-6903 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.0.0, 2.4.1, 2.6.0 > Environment: PowerPC RHEL 7 & 6.5 ( ppc64 - Big-Endian ) >Reporter: Ayappan >Priority: Blocker > > Native Crc32 checksum calculation is not handled in Big-Endian > Architecture.In this case, the platform is ppc64. Due to this several > testcases in HDFS module fails. > Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum > Tests run: 3, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 13.274 sec > <<< FAILURE! - in org.apache.hadoop.hdfs.TestAppendDifferentChecksum > testAlgoSwitchRandomized(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 7.141 sec <<< ERROR! > java.io.IOException: p=/testAlgoSwitchRandomized, length=28691, i=12288 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testAlgoSwitchRandomized(TestAppendDifferentChecksum.java:130) > testSwitchAlgorithms(org.apache.hadoop.hdfs.TestAppendDifferentChecksum) > Time elapsed: 1.394 sec <<< ERROR! > java.io.IOException: p=/testSwitchAlgorithms, length=3000, i=0 > at org.apache.hadoop.util.NativeCrc32.nativeVerifyChunkedSums(Native > Method) > at > org.apache.hadoop.util.NativeCrc32.verifyChunkedSums(NativeCrc32.java:57) > at > org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:291) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.readNextPacket(RemoteBlockReader2.java:202) > at > org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.java:137) > at > org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:682) > at > org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:738) > at > org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:795) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:836) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:644) > at java.io.FilterInputStream.read(FilterInputStream.java:83) > at > org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:129) > at > org.apache.hadoop.hdfs.TestAppendDifferentChecksum.testSwitchAlgorithms(TestAppendDifferentChecksum.java:94) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6904) YARN unable to renew delegation token fetched via webhdfs due to incorrect service port
[
https://issues.apache.org/jira/browse/HDFS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated HDFS-6904:
Description:
YARN is unable to renew delegation tokens obtained via the WebHDFS REST API.
The scenario is as follows -
1. User creates a delegation token using the WebHDFS REST API
2. User passes this token to YARN as part of app submission(via the YARN REST
API)
3. When YARN tries to renew this delegation token, it fails because the token
service is pointing to the RPC port but the token kind is WebHDFS.
The exception is
{noformat}
2014-08-19 03:12:54,733 WARN security.DelegationTokenRenewer
(DelegationTokenRenewer.java:handleDTRenewerAppSubmitEvent(661)) - Unable to
add the application to the delegation token renewer.
java.io.IOException: Failed to renew token: Kind: WEBHDFS delegation, Service:
NameNodeIP:8020, Ident: (WEBHDFS delegation token for hrt_qa)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:394)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$5(DelegationTokenRenewer.java:357)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unexpected HTTP response: code=-1 != 200,
op=RENEWDELEGATIONTOKEN, message=null
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:331)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:90)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:598)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:448)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:477)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:473)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.renewDelegationToken(WebHdfsFileSystem.java:1318)
at
org.apache.hadoop.hdfs.web.TokenAspect$TokenManager.renew(TokenAspect.java:73)
at org.apache.hadoop.security.token.Token.renew(Token.java:377)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:477)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.renewToken(DelegationTokenRenewer.java:473)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:392)
... 6 more
Caused by: java.io.IOException: The error stream is null.
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:304)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:329)
... 24 more
2014-08-19 03:12:54,735 DEBUG event.AsyncDispatcher
(AsyncDispatcher.java:dispatch(164)) - Dispatching the event
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppRejectedEvent.EventType:
APP_REJECTED
{noformat}
I suspect the issue is that the Namenode generates a delegation token of kind
WebHDFS but doesn't change the service port. When YARN tries to renew the
delegation token, it ends up trying to contact WebHDFS on the RPC port.
>From NamenodeWebHdfsMethods.java
{noformat}
case GETDELEGATIONTOKEN:
{
if (delegation.getValue() != null) {
throw new IllegalArgumentException(delegation.getName()
+ " parameter is not null.");
}
final Token token = generateDelegationToken(
namenode, ugi, renewer.getValue());
final String js = JsonUtil.toJsonString(token);
return Response.ok(js
[jira] [Created] (HDFS-6906) Archival Storage: Add more tests for BlockStoragePolicy
Tsz Wo Nicholas Sze created HDFS-6906: - Summary: Archival Storage: Add more tests for BlockStoragePolicy Key: HDFS-6906 URL: https://issues.apache.org/jira/browse/HDFS-6906 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Priority: Minor The logic for choosing storage types is tricky, especially when there are chosen storage types and/or unavailable storage types. Let's add more unit tests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6905) fs-encryption merge triggered released audit failures
Allen Wittenauer created HDFS-6905: -- Summary: fs-encryption merge triggered released audit failures Key: HDFS-6905 URL: https://issues.apache.org/jira/browse/HDFS-6905 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Allen Wittenauer Priority: Blocker Release audit is failing on three files since the merge of fs-encryption code due to missing Apache license: * hdfs/protocol/EncryptionZoneWithId.java * hdfs/server/namenode/EncryptionFaultInjector.java * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-6905: --- Summary: fs-encryption merge triggered release audit failures (was: fs-encryption merge triggered released audit failures) > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Priority: Blocker > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-6905: --- Labels: newbie (was: ) > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Priority: Blocker > Labels: newbie > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6906) Archival Storage: Add more tests for BlockStoragePolicy
[ https://issues.apache.org/jira/browse/HDFS-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-6906: -- Attachment: h6906_20140821.patch h6906_20140821.patch: adds more tests. > Archival Storage: Add more tests for BlockStoragePolicy > --- > > Key: HDFS-6906 > URL: https://issues.apache.org/jira/browse/HDFS-6906 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Attachments: h6906_20140821.patch > > > The logic for choosing storage types is tricky, especially when there are > chosen storage types and/or unavailable storage types. Let's add more unit > tests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb reassigned HDFS-6905: -- Assignee: Charles Lamb > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Charles Lamb >Priority: Blocker > Labels: newbie > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6905 started by Charles Lamb. > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Charles Lamb >Priority: Blocker > Labels: newbie > Attachments: HDFS-6905.001.patch > > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6905: --- Attachment: HDFS-6905.001.patch Attached is a patch which adds the required headers. > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Charles Lamb >Priority: Blocker > Labels: newbie > Attachments: HDFS-6905.001.patch > > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6905: --- Status: Patch Available (was: In Progress) > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Charles Lamb >Priority: Blocker > Labels: newbie > Attachments: HDFS-6905.001.patch > > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105494#comment-14105494 ] Yi Liu commented on HDFS-6905: -- Right, there files miss Apache license. >From the Jenkins result of the merged patch in HDFS-6134, we have not seen >these, seems Jenkins sometime can't report all issues if the patch gets too >large? > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Charles Lamb >Priority: Blocker > Labels: newbie > Attachments: HDFS-6905.001.patch > > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105490#comment-14105490 ] Arpit Agarwal commented on HDFS-6581: - Created a development branch HDFS-6581 per r1619458. > Write to single replica in memory > - > > Key: HDFS-6581 > URL: https://issues.apache.org/jira/browse/HDFS-6581 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFSWriteableReplicasInMemory.pdf > > > Per discussion with the community on HDFS-5851, we will implement writing to > a single replica in DN memory via DataTransferProtocol. > This avoids some of the issues with short-circuit writes, which we can > revisit at a later time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6905) fs-encryption merge triggered release audit failures
[ https://issues.apache.org/jira/browse/HDFS-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105502#comment-14105502 ] Allen Wittenauer commented on HDFS-6905: I'm not sure what's going on with Jenkins lately. It was stable after the move for a bit, then it started throwing weird errors again. Saw these failures in another jenkins run through. So there's that I guess? I'm really curious as to what it will say with this patch. :D > fs-encryption merge triggered release audit failures > > > Key: HDFS-6905 > URL: https://issues.apache.org/jira/browse/HDFS-6905 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Charles Lamb >Priority: Blocker > Labels: newbie > Attachments: HDFS-6905.001.patch > > > Release audit is failing on three files since the merge of fs-encryption code > due to missing Apache license: > * hdfs/protocol/EncryptionZoneWithId.java > * hdfs/server/namenode/EncryptionFaultInjector.java > * hdfs/server/namenode/EncryptionZoneManager.java -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Attachment: HDFS-6865.4.patch Reattching same patch to retrigger Jenkins. > Byte array native checksumming on client side (HDFS changes) > > > Key: HDFS-6865 > URL: https://issues.apache.org/jira/browse/HDFS-6865 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, performance >Reporter: James Thomas >Assignee: James Thomas > Attachments: HDFS-6865.2.patch, HDFS-6865.3.patch, HDFS-6865.4.patch, > HDFS-6865.patch > > > Refactor FSOutputSummer to buffer data and use the native checksum > calculation functionality introduced in HADOOP-10975. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6870) Blocks and INodes could leak for Rename with overwrite flag
[
https://issues.apache.org/jira/browse/HDFS-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105554#comment-14105554
]
Hudson commented on HDFS-6870:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #653 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/653/])
HDFS-6870. Blocks and INodes could leak for Rename with overwrite flag.
Contributed by Yi Liu. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619192)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRename.java
> Blocks and INodes could leak for Rename with overwrite flag
> ---
>
> Key: HDFS-6870
> URL: https://issues.apache.org/jira/browse/HDFS-6870
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: 2.6.0
>
> Attachments: HDFS-6870.001.patch
>
>
> Following code in FSDirectory#unprotectedRenameTo doesn't collect blocks and
> INodes for non snapshot path.
> {code}
> if (removedDst != null) {
> undoRemoveDst = false;
> if (removedNum > 0) {
> BlocksMapUpdateInfo collectedBlocks = new BlocksMapUpdateInfo();
> List removedINodes = new ChunkedArrayList();
> filesDeleted = removedDst.cleanSubtree(Snapshot.CURRENT_STATE_ID,
> dstIIP.getLatestSnapshotId(), collectedBlocks, removedINodes,
> true).get(Quota.NAMESPACE);
> getFSNamesystem().removePathAndBlocks(src, collectedBlocks,
> removedINodes, false);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6858) Allow dfs.data.transfer.saslproperties.resolver.class default to hadoop.security.saslproperties.resolver.class
[
https://issues.apache.org/jira/browse/HDFS-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105558#comment-14105558
]
Hudson commented on HDFS-6858:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #653 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/653/])
HDFS-6858. Allow dfs.data.transfer.saslproperties.resolver.class default to
hadoop.security.saslproperties.resolver.class. Contributed by Benoy Antony.
(cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619256)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/DataTransferSaslUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Allow dfs.data.transfer.saslproperties.resolver.class default to
> hadoop.security.saslproperties.resolver.class
> --
>
> Key: HDFS-6858
> URL: https://issues.apache.org/jira/browse/HDFS-6858
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.5.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
>Priority: Minor
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6858.patch, HDFS-6858.patch
>
>
> Currently administrator needs to configure both
> {code}dfs.data.transfer.saslproperties.resolver.class{code} and
> {code}hadoop.security.saslproperties.resolver.class{code} to override
> saslproperties for rpc and data transfer properties. In some cases, the
> values for these two properties will be same. In those cases, it may be
> easier if {code}dfs.data.transfer.saslproperties.resolver.class{code}
> defaults to {code}hadoop.security.saslproperties.resolver.class{code}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105557#comment-14105557 ] Hudson commented on HDFS-6134: -- FAILURE: Integrated in Hadoop-Yarn-trunk #653 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/653/]) Fix up CHANGES.txt for HDFS-6134, HADOOP-10150 and related JIRAs. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619203) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt HDFS-6134 and HADOOP-10150 subtasks. Merge fs-encryption branch to trunk. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619197) * /hadoop/common/trunk * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/config.h.cmake * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/AesCtrCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CipherSuite.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/Decryptor.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/Encryptor.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/JceAesCtrCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslAesCtrCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslCipher.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataOutputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileEncryptionInfo.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/crypto * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/CryptoStreamsTestBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/a
[jira] [Commented] (HDFS-6878) Change MiniDFSCluster to support StorageType configuration for individual directories
[ https://issues.apache.org/jira/browse/HDFS-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105559#comment-14105559 ] Hudson commented on HDFS-6878: -- FAILURE: Integrated in Hadoop-Yarn-trunk #653 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/653/]) HDFS-6878. Change MiniDFSCluster to support StorageType configuration for individual directories. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619271) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSClusterWithNodeGroup.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStorageReport.java > Change MiniDFSCluster to support StorageType configuration for individual > directories > - > > Key: HDFS-6878 > URL: https://issues.apache.org/jira/browse/HDFS-6878 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Arpit Agarwal >Priority: Minor > Fix For: 3.0.0, 2.6.0 > > Attachments: HDFS-6878.01.patch, HDFS-6878.02.patch, > HDFS-6878.03.patch > > > Currently, MiniDFSCluster only supports a single StorageType configuration > for all datadnodes, i.e. setting all directories in all datanodes to one > StorageType. It should support setting individual StorageType for each > directory in each datanode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6758) block writer should pass the expected block size to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105561#comment-14105561
]
Hudson commented on HDFS-6758:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #653 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/653/])
HDFS-6758. Block writer should pass the expected block size to
DataXceiverServer (Arpit Agarwal) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619275)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java
> block writer should pass the expected block size to DataXceiverServer
> -
>
> Key: HDFS-6758
> URL: https://issues.apache.org/jira/browse/HDFS-6758
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client
>Affects Versions: 2.4.1
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6758.01.patch, HDFS-6758.02.patch
>
>
> DataXceiver initializes the block size to the default block size for the
> cluster. This size is later used by the FsDatasetImpl when applying
> VolumeChoosingPolicy.
> {code}
> block.setNumBytes(dataXceiverServer.estimateBlockSize);
> {code}
> where
> {code}
> /**
>* We need an estimate for block size to check if the disk partition has
>* enough space. For now we set it to be the default block size set
>* in the server side configuration, which is not ideal because the
>* default block size should be a client-size configuration.
>* A better solution is to include in the header the estimated block size,
>* i.e. either the actual block size or the default block size.
>*/
> final long estimateBlockSize;
> {code}
> In most cases the writer can just pass the maximum expected block size to the
> DN instead of having to use the cluster default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6376) Distcp data between two HA clusters requires another configuration
[ https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105586#comment-14105586 ] Dave Marion commented on HDFS-6376: --- Thanks for reviewing this. I thought this was dead and that I would forever have to patch Hadoop for our application. Out of curiosity, have others started running into this issue? bq. have you tested your patch for distcp between real clusters? Yes. I have been running a version of this patch for about 2 months on a test cluster. We are using Hadoop 2 so the patch that I am applying is a little different. My Hadoop 2 patch also includes a change in the dfsclusterhealth.jsp file so that only NameNodes in "this" cluster are shown. I could not find the same jsp file in the Hadoop 3 source. Generally speaking, I think I have fixed all locations in the code that need to be fixed, but I could be missing something that I don't know about. As you can see from the patch history, I thought I had to make a change DFSUtil, but it broke some things and I had to revert those changes. bq. It will be great if you can generally mention how you patch works for both secured and insecure HA clusters. We are not using secured HA, it has not been tested in that manner bq. Another nit is that we need to fix indents in the new unit test. I'm happy to fix. bq. Maybe we can rename the new configuration from "dfs.nameservice.cluster.excludes" to something like "dfs.nameservices.cluster.outside" I have no issues with changing the name. In my situation I have multiple HDFS nameservices defined in hdfs-site.xml and I want to explicitly state which ones are not part of "this" cluster. Exclude seemed like a good term for that. > Distcp data between two HA clusters requires another configuration > -- > > Key: HDFS-6376 > URL: https://issues.apache.org/jira/browse/HDFS-6376 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, federation, hdfs-client >Affects Versions: 2.3.0, 2.4.0 > Environment: Hadoop 2.3.0 >Reporter: Dave Marion >Assignee: Dave Marion > Fix For: 3.0.0 > > Attachments: HDFS-6376-2.patch, HDFS-6376-3-branch-2.4.patch, > HDFS-6376-4-branch-2.4.patch, HDFS-6376-5-trunk.patch, > HDFS-6376-6-trunk.patch, HDFS-6376-7-trunk.patch, HDFS-6376-branch-2.4.patch, > HDFS-6376-patch-1.patch > > > User has to create a third set of configuration files for distcp when > transferring data between two HA clusters. > Consider the scenario in [1]. You cannot put all of the required properties > in core-site.xml and hdfs-site.xml for the client to resolve the location of > both active namenodes. If you do, then the datanodes from cluster A may join > cluster B. I can not find a configuration option that tells the datanodes to > federate blocks for only one of the clusters in the configuration. > [1] > http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105604#comment-14105604
]
Kihwal Lee commented on HDFS-6888:
--
See the patch in HDFS-3733. When this logging was added, a number of test cases
were added. These test cases need to be modified. "open" (i.e.
getBlockLocations()) is logged for webhdfs. For example,
{{testAuditWebHdfsOpen()} generates 3 audit log entries: getfileinfo/webhdfs,
open/webhdfs and open/rpc. After this change, it will have the last two
entries.
> Remove audit logging of getFIleInfo()
> -
>
> Key: HDFS-6888
> URL: https://issues.apache.org/jira/browse/HDFS-6888
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Kihwal Lee
>Assignee: Chen He
> Labels: log
> Attachments: HDFS-6888.patch
>
>
> The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
> one of the most called method, users have noticed that audit log is now
> filled with this. Since we now have HTTP request logging, this seems
> unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105623#comment-14105623 ] Gera Shegalov commented on HDFS-6888: - getfileinfo a common RPC but we have identified many times using the audit log an app that needs some "help" in optimizing their FileSystem API usage. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6581: Attachment: HDFS-6581-prototype.01.patch A patch to demonstrate the approach we've been trying out. This is a somewhat refined prototype but there are many rough edges we'll work on in the private branch, specifically around the lazy writer, eviction and recovery on restart. > Write to single replica in memory > - > > Key: HDFS-6581 > URL: https://issues.apache.org/jira/browse/HDFS-6581 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6581-prototype.01.patch, > HDFSWriteableReplicasInMemory.pdf > > > Per discussion with the community on HDFS-5851, we will implement writing to > a single replica in DN memory via DataTransferProtocol. > This avoids some of the issues with short-circuit writes, which we can > revisit at a later time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105639#comment-14105639 ] Arpit Agarwal edited comment on HDFS-6581 at 8/21/14 5:30 PM: -- A patch to demonstrate the approach we've been trying out. This is a somewhat refined prototype but there are many rough edges we'll work on in the feature branch, specifically around the lazy writer, eviction and recovery on restart. was (Author: arpitagarwal): A patch to demonstrate the approach we've been trying out. This is a somewhat refined prototype but there are many rough edges we'll work on in the private branch, specifically around the lazy writer, eviction and recovery on restart. > Write to single replica in memory > - > > Key: HDFS-6581 > URL: https://issues.apache.org/jira/browse/HDFS-6581 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6581-prototype.01.patch, > HDFSWriteableReplicasInMemory.pdf > > > Per discussion with the community on HDFS-5851, we will implement writing to > a single replica in DN memory via DataTransferProtocol. > This avoids some of the issues with short-circuit writes, which we can > revisit at a later time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6581: Status: Patch Available (was: In Progress) > Write to single replica in memory > - > > Key: HDFS-6581 > URL: https://issues.apache.org/jira/browse/HDFS-6581 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6581-prototype.01.patch, > HDFSWriteableReplicasInMemory.pdf > > > Per discussion with the community on HDFS-5851, we will implement writing to > a single replica in DN memory via DataTransferProtocol. > This avoids some of the issues with short-circuit writes, which we can > revisit at a later time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6863) Archival Storage: Support migration for snapshot paths
[ https://issues.apache.org/jira/browse/HDFS-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6863: Attachment: HDFS-6863.001.patch Thanks for the review, [~szetszwo]! You're right, the check should only be on files. Update the patch to fix this. I will add more unit tests later. > Archival Storage: Support migration for snapshot paths > -- > > Key: HDFS-6863 > URL: https://issues.apache.org/jira/browse/HDFS-6863 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-6863.000.patch, HDFS-6863.001.patch > > > Per discussion in HDFS-6801, we use this jira to support migrate files and > directories that only exist in snapshots (i.e., files/dirs that have been > deleted from the current fsdir). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6890) NFS readdirplus doesn't return dotdot attributes
[
https://issues.apache.org/jira/browse/HDFS-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105675#comment-14105675
]
Brandon Li commented on HDFS-6890:
--
The release audit warnings are not introduced by this patch. HDFS-6905 has been
filed for the release audit issue.
> NFS readdirplus doesn't return dotdot attributes
>
>
> Key: HDFS-6890
> URL: https://issues.apache.org/jira/browse/HDFS-6890
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-6890.001.patch
>
>
> In RpcProgramNfs3#readdirplus():
> {noformat}
> entries[1] = new READDIRPLUS3Response.EntryPlus3(dotdotFileId, "..",
> dotdotFileId, postOpDirAttr, new FileHandle(dotdotFileId));
> {noformat}
> It should return the directory's parent attribute instead of postOpDirAttr.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6898: Status: Patch Available (was: In Progress) > DN must reserve space for a full block when an RBW block is created > --- > > Key: HDFS-6898 > URL: https://issues.apache.org/jira/browse/HDFS-6898 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Gopal V >Assignee: Arpit Agarwal > Attachments: HDFS-6898-with-6899.02.patch, HDFS-6898.01.patch > > > DN will successfully create two RBW blocks on the same volume even if the > free space is sufficient for just one full block. > One or both block writers may subsequently get a DiskOutOfSpace exception. > This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6898: Attachment: HDFS-6898-with-6899.02.patch > DN must reserve space for a full block when an RBW block is created > --- > > Key: HDFS-6898 > URL: https://issues.apache.org/jira/browse/HDFS-6898 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Gopal V >Assignee: Arpit Agarwal > Attachments: HDFS-6898-with-6899.02.patch, HDFS-6898.01.patch > > > DN will successfully create two RBW blocks on the same volume even if the > free space is sufficient for just one full block. > One or both block writers may subsequently get a DiskOutOfSpace exception. > This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6907) Source files missing license headers
Arpit Agarwal created HDFS-6907:
---
Summary: Source files missing license headers
Key: HDFS-6907
URL: https://issues.apache.org/jira/browse/HDFS-6907
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0, 2.6.0
Reporter: Arpit Agarwal
The following files were committed without license headers as flagged by
Jenkins.
{code}
!?
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionFaultInjector.java
!?
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionZoneManager.java
!?
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneWithId.java
Lines that start with ? in the release audit report indicate files that do
not have an Apache license header.
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6899) Allow changing MiniDFSCluster volumes per DN and capacity per volume
[ https://issues.apache.org/jira/browse/HDFS-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105682#comment-14105682 ] Arpit Agarwal commented on HDFS-6899: - Audit warnings are unrelated to the patch, filed HDFS-6907. > Allow changing MiniDFSCluster volumes per DN and capacity per volume > > > Key: HDFS-6899 > URL: https://issues.apache.org/jira/browse/HDFS-6899 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, test >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6899.01.patch > > > MiniDFSCluster hardcodes the number of directories per volume to two. Propose > removing the hard-coded restriction. > It would be useful to limit the capacity of individual storage directories > for testing purposes. There is already a way to do so for SimulatedFSDataset, > we can add one when using real volumes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6890) NFS readdirplus doesn't return dotdot attributes
[
https://issues.apache.org/jira/browse/HDFS-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105689#comment-14105689
]
Brandon Li commented on HDFS-6890:
--
I've committed the patch. Thank you, [~jingzhao].
> NFS readdirplus doesn't return dotdot attributes
>
>
> Key: HDFS-6890
> URL: https://issues.apache.org/jira/browse/HDFS-6890
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Fix For: 2.6.0
>
> Attachments: HDFS-6890.001.patch
>
>
> In RpcProgramNfs3#readdirplus():
> {noformat}
> entries[1] = new READDIRPLUS3Response.EntryPlus3(dotdotFileId, "..",
> dotdotFileId, postOpDirAttr, new FileHandle(dotdotFileId));
> {noformat}
> It should return the directory's parent attribute instead of postOpDirAttr.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6890) NFS readdirplus doesn't return dotdot attributes
[
https://issues.apache.org/jira/browse/HDFS-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-6890:
-
Fix Version/s: 2.6.0
> NFS readdirplus doesn't return dotdot attributes
>
>
> Key: HDFS-6890
> URL: https://issues.apache.org/jira/browse/HDFS-6890
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Fix For: 2.6.0
>
> Attachments: HDFS-6890.001.patch
>
>
> In RpcProgramNfs3#readdirplus():
> {noformat}
> entries[1] = new READDIRPLUS3Response.EntryPlus3(dotdotFileId, "..",
> dotdotFileId, postOpDirAttr, new FileHandle(dotdotFileId));
> {noformat}
> It should return the directory's parent attribute instead of postOpDirAttr.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6890) NFS readdirplus doesn't return dotdot attributes
[
https://issues.apache.org/jira/browse/HDFS-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-6890:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> NFS readdirplus doesn't return dotdot attributes
>
>
> Key: HDFS-6890
> URL: https://issues.apache.org/jira/browse/HDFS-6890
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-6890.001.patch
>
>
> In RpcProgramNfs3#readdirplus():
> {noformat}
> entries[1] = new READDIRPLUS3Response.EntryPlus3(dotdotFileId, "..",
> dotdotFileId, postOpDirAttr, new FileHandle(dotdotFileId));
> {noformat}
> It should return the directory's parent attribute instead of postOpDirAttr.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105694#comment-14105694 ] Konstantin Shvachko commented on HDFS-6134: --- Guys, looks like you ignored javac compiler warnings from Jenkins. I see unused members and imports in FSNamesystem after the merge. May be you can fix it in some follow up jira, rather than creating a separate one. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 3.0.0, 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Fix For: 3.0.0 > > Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, > HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf, > fs-encryption.2014-08-18.patch, fs-encryption.2014-08-19.patch > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105698#comment-14105698 ] Kihwal Lee commented on HDFS-6888: -- What if we log it at debug level? > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6907) Source files missing license headers
[
https://issues.apache.org/jira/browse/HDFS-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved HDFS-6907.
--
Resolution: Duplicate
Dup of HDFS-6905.
> Source files missing license headers
>
>
> Key: HDFS-6907
> URL: https://issues.apache.org/jira/browse/HDFS-6907
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Arpit Agarwal
>
> The following files were committed without license headers as flagged by
> Jenkins.
> {code}
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionFaultInjector.java
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionZoneManager.java
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneWithId.java
> Lines that start with ? in the release audit report indicate files that
> do not have an Apache license header.
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5135) Umbrella JIRA for NFS end to end unit test frameworks
[ https://issues.apache.org/jira/browse/HDFS-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105717#comment-14105717 ] Brandon Li commented on HDFS-5135: -- TestOutOfOrderWrite was created when AUTH_NONE was supported, but this test is not updated along with other code changes. The test won't pass with the current NFS implementation since we don't support AUTH_NONE anymore. > Umbrella JIRA for NFS end to end unit test frameworks > - > > Key: HDFS-5135 > URL: https://issues.apache.org/jira/browse/HDFS-5135 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.2.0 >Reporter: Brandon Li >Assignee: Zhe Zhang > > Currently, we have to manually start portmap and nfs3 processes to test patch > and new functionalities. This JIRA is to track the effort to introduce a test > framework to NFS unit test without starting standalone nfs3 processes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105722#comment-14105722
]
Kihwal Lee commented on HDFS-6888:
--
We could have {{logAuditEvent()}} check {{cmd}} against "getfileinfo" or a
collection of such commands and log at debug level. Otherwise we will have to
change HdfsAuditLogger, which will introduce incompatibility. Users who want
to see this in the log can change the log level at runtime.
> Remove audit logging of getFIleInfo()
> -
>
> Key: HDFS-6888
> URL: https://issues.apache.org/jira/browse/HDFS-6888
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Kihwal Lee
>Assignee: Chen He
> Labels: log
> Attachments: HDFS-6888.patch
>
>
> The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
> one of the most called method, users have noticed that audit log is now
> filled with this. Since we now have HTTP request logging, this seems
> unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6899) Allow changing MiniDFSCluster volumes per DN and capacity per volume
[ https://issues.apache.org/jira/browse/HDFS-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105728#comment-14105728 ] Jing Zhao commented on HDFS-6899: - Thanks for working on this, [~arpitagarwal]! The patch looks good to me. But do you think it is possible to add a data field (long[][]) and corresponding setter in MiniDFSCluster which allows users to set the capacity for each volume of each DataNode? > Allow changing MiniDFSCluster volumes per DN and capacity per volume > > > Key: HDFS-6899 > URL: https://issues.apache.org/jira/browse/HDFS-6899 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, test >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6899.01.patch > > > MiniDFSCluster hardcodes the number of directories per volume to two. Propose > removing the hard-coded restriction. > It would be useful to limit the capacity of individual storage directories > for testing purposes. There is already a way to do so for SimulatedFSDataset, > we can add one when using real volumes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3689) Add support for variable length block
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105733#comment-14105733 ] Colin Patrick McCabe commented on HDFS-3689: bq. One thought is that we could identify runs of zeros fairly easily by looking at the checksums: an all-zero checksum chunk has a constant crc32 which we can compare for in a single instruction. The DN could relatively easily loop through the checksums of an incoming data packet, and verify whether it is all zeros, and if so, turn it into a sparse write. Interesting idea. This would allow us to automatically deal with long stretches of zeroes by creating sparse block files on the datanode. Of course we have to check that the zero checksum really did come from a zeroed checksum chunk, rather than an unlikely coincidence. I wonder if we could create sparse files without any new APIs this way... > Add support for variable length block > - > > Key: HDFS-3689 > URL: https://issues.apache.org/jira/browse/HDFS-3689 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs-client, namenode >Affects Versions: 3.0.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch > > > Currently HDFS supports fixed length blocks. Supporting variable length block > will allow new use cases and features to be built on top of HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4486) Add log category for long-running DFSClient notices
[
https://issues.apache.org/jira/browse/HDFS-4486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105738#comment-14105738
]
Colin Patrick McCabe commented on HDFS-4486:
Looks good. Can we also add this to {{JniBasedGroupsMappingWithFallback}}, to
the encryption classes that have a fallback (if openssl is not found), and to
the short-circuit fallback messages
> Add log category for long-running DFSClient notices
> ---
>
> Key: HDFS-4486
> URL: https://issues.apache.org/jira/browse/HDFS-4486
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Todd Lipcon
>Assignee: Zhe Zhang
>Priority: Minor
> Attachments: HDFS-4486-20140820.patch
>
>
> There are a number of features in the DFS client which are transparent but
> can make a fairly big difference for performance -- two in particular are
> short circuit reads and native checksumming. Because we don't want log spew
> for clients like "hadoop fs -cat" we currently log only at DEBUG level when
> these features are disabled. This makes it difficult to troubleshoot/verify
> for long-running perf-sensitive clients like HBase.
> One simple solution is to add a new log category - eg
> o.a.h.h.DFSClient.PerformanceAdvisory - which long-running clients could
> enable at DEBUG level without getting the full debug spew.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5135) Umbrella JIRA for NFS end to end unit test frameworks
[ https://issues.apache.org/jira/browse/HDFS-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105753#comment-14105753 ] Zhe Zhang commented on HDFS-5135: - I see, thanks for the clarification. I just came across another issue in the code: if we send a CREATE3Request to the NFS gateway, the received RpcReply can't be correctly parsed by RpcReply.read(). Somehow xid becomes the second int instead of the first. [~brandonli] Do you think this is a bug in the RpcReply.read() method or the code that generates and sends the RPC reply? > Umbrella JIRA for NFS end to end unit test frameworks > - > > Key: HDFS-5135 > URL: https://issues.apache.org/jira/browse/HDFS-5135 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.2.0 >Reporter: Brandon Li >Assignee: Zhe Zhang > > Currently, we have to manually start portmap and nfs3 processes to test patch > and new functionalities. This JIRA is to track the effort to introduce a test > framework to NFS unit test without starting standalone nfs3 processes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6758) block writer should pass the expected block size to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105783#comment-14105783
]
Hudson commented on HDFS-6758:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1844 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1844/])
HDFS-6758. Block writer should pass the expected block size to
DataXceiverServer (Arpit Agarwal) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619275)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java
> block writer should pass the expected block size to DataXceiverServer
> -
>
> Key: HDFS-6758
> URL: https://issues.apache.org/jira/browse/HDFS-6758
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client
>Affects Versions: 2.4.1
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6758.01.patch, HDFS-6758.02.patch
>
>
> DataXceiver initializes the block size to the default block size for the
> cluster. This size is later used by the FsDatasetImpl when applying
> VolumeChoosingPolicy.
> {code}
> block.setNumBytes(dataXceiverServer.estimateBlockSize);
> {code}
> where
> {code}
> /**
>* We need an estimate for block size to check if the disk partition has
>* enough space. For now we set it to be the default block size set
>* in the server side configuration, which is not ideal because the
>* default block size should be a client-size configuration.
>* A better solution is to include in the header the estimated block size,
>* i.e. either the actual block size or the default block size.
>*/
> final long estimateBlockSize;
> {code}
> In most cases the writer can just pass the maximum expected block size to the
> DN instead of having to use the cluster default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6890) NFS readdirplus doesn't return dotdot attributes
[
https://issues.apache.org/jira/browse/HDFS-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105775#comment-14105775
]
Hudson commented on HDFS-6890:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6092 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6092/])
HDFS-6890. NFS readdirplus doesn't return dotdot attributes. Contributed by
Brandon Li (brandonli:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619500)
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NFS readdirplus doesn't return dotdot attributes
>
>
> Key: HDFS-6890
> URL: https://issues.apache.org/jira/browse/HDFS-6890
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.2.0
>Reporter: Brandon Li
>Assignee: Brandon Li
> Fix For: 2.6.0
>
> Attachments: HDFS-6890.001.patch
>
>
> In RpcProgramNfs3#readdirplus():
> {noformat}
> entries[1] = new READDIRPLUS3Response.EntryPlus3(dotdotFileId, "..",
> dotdotFileId, postOpDirAttr, new FileHandle(dotdotFileId));
> {noformat}
> It should return the directory's parent attribute instead of postOpDirAttr.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6878) Change MiniDFSCluster to support StorageType configuration for individual directories
[ https://issues.apache.org/jira/browse/HDFS-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105781#comment-14105781 ] Hudson commented on HDFS-6878: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1844 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1844/]) HDFS-6878. Change MiniDFSCluster to support StorageType configuration for individual directories. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619271) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSClusterWithNodeGroup.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStorageReport.java > Change MiniDFSCluster to support StorageType configuration for individual > directories > - > > Key: HDFS-6878 > URL: https://issues.apache.org/jira/browse/HDFS-6878 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Arpit Agarwal >Priority: Minor > Fix For: 3.0.0, 2.6.0 > > Attachments: HDFS-6878.01.patch, HDFS-6878.02.patch, > HDFS-6878.03.patch > > > Currently, MiniDFSCluster only supports a single StorageType configuration > for all datadnodes, i.e. setting all directories in all datanodes to one > StorageType. It should support setting individual StorageType for each > directory in each datanode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105779#comment-14105779 ] Hudson commented on HDFS-6134: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1844 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1844/]) Fix up CHANGES.txt for HDFS-6134, HADOOP-10150 and related JIRAs. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619203) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt HDFS-6134 and HADOOP-10150 subtasks. Merge fs-encryption branch to trunk. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619197) * /hadoop/common/trunk * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/config.h.cmake * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/AesCtrCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CipherSuite.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/Decryptor.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/Encryptor.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/JceAesCtrCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslAesCtrCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslCipher.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataOutputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileEncryptionInfo.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/crypto * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/CryptoStreamsTestBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org
[jira] [Commented] (HDFS-6870) Blocks and INodes could leak for Rename with overwrite flag
[
https://issues.apache.org/jira/browse/HDFS-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105776#comment-14105776
]
Hudson commented on HDFS-6870:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1844 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1844/])
HDFS-6870. Blocks and INodes could leak for Rename with overwrite flag.
Contributed by Yi Liu. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619192)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRename.java
> Blocks and INodes could leak for Rename with overwrite flag
> ---
>
> Key: HDFS-6870
> URL: https://issues.apache.org/jira/browse/HDFS-6870
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: 2.6.0
>
> Attachments: HDFS-6870.001.patch
>
>
> Following code in FSDirectory#unprotectedRenameTo doesn't collect blocks and
> INodes for non snapshot path.
> {code}
> if (removedDst != null) {
> undoRemoveDst = false;
> if (removedNum > 0) {
> BlocksMapUpdateInfo collectedBlocks = new BlocksMapUpdateInfo();
> List removedINodes = new ChunkedArrayList();
> filesDeleted = removedDst.cleanSubtree(Snapshot.CURRENT_STATE_ID,
> dstIIP.getLatestSnapshotId(), collectedBlocks, removedINodes,
> true).get(Quota.NAMESPACE);
> getFSNamesystem().removePathAndBlocks(src, collectedBlocks,
> removedINodes, false);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6858) Allow dfs.data.transfer.saslproperties.resolver.class default to hadoop.security.saslproperties.resolver.class
[
https://issues.apache.org/jira/browse/HDFS-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105780#comment-14105780
]
Hudson commented on HDFS-6858:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1844 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1844/])
HDFS-6858. Allow dfs.data.transfer.saslproperties.resolver.class default to
hadoop.security.saslproperties.resolver.class. Contributed by Benoy Antony.
(cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619256)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/DataTransferSaslUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Allow dfs.data.transfer.saslproperties.resolver.class default to
> hadoop.security.saslproperties.resolver.class
> --
>
> Key: HDFS-6858
> URL: https://issues.apache.org/jira/browse/HDFS-6858
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.5.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
>Priority: Minor
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6858.patch, HDFS-6858.patch
>
>
> Currently administrator needs to configure both
> {code}dfs.data.transfer.saslproperties.resolver.class{code} and
> {code}hadoop.security.saslproperties.resolver.class{code} to override
> saslproperties for rpc and data transfer properties. In some cases, the
> values for these two properties will be same. In those cases, it may be
> easier if {code}dfs.data.transfer.saslproperties.resolver.class{code}
> defaults to {code}hadoop.security.saslproperties.resolver.class{code}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6870) Blocks and INodes could leak for Rename with overwrite flag
[
https://issues.apache.org/jira/browse/HDFS-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105844#comment-14105844
]
Hudson commented on HDFS-6870:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1870 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1870/])
HDFS-6870. Blocks and INodes could leak for Rename with overwrite flag.
Contributed by Yi Liu. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619192)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSRename.java
> Blocks and INodes could leak for Rename with overwrite flag
> ---
>
> Key: HDFS-6870
> URL: https://issues.apache.org/jira/browse/HDFS-6870
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: 2.6.0
>
> Attachments: HDFS-6870.001.patch
>
>
> Following code in FSDirectory#unprotectedRenameTo doesn't collect blocks and
> INodes for non snapshot path.
> {code}
> if (removedDst != null) {
> undoRemoveDst = false;
> if (removedNum > 0) {
> BlocksMapUpdateInfo collectedBlocks = new BlocksMapUpdateInfo();
> List removedINodes = new ChunkedArrayList();
> filesDeleted = removedDst.cleanSubtree(Snapshot.CURRENT_STATE_ID,
> dstIIP.getLatestSnapshotId(), collectedBlocks, removedINodes,
> true).get(Quota.NAMESPACE);
> getFSNamesystem().removePathAndBlocks(src, collectedBlocks,
> removedINodes, false);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6758) block writer should pass the expected block size to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105851#comment-14105851
]
Hudson commented on HDFS-6758:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1870 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1870/])
HDFS-6758. Block writer should pass the expected block size to
DataXceiverServer (Arpit Agarwal) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619275)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteBlockGetsBlockLengthHint.java
> block writer should pass the expected block size to DataXceiverServer
> -
>
> Key: HDFS-6758
> URL: https://issues.apache.org/jira/browse/HDFS-6758
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs-client
>Affects Versions: 2.4.1
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6758.01.patch, HDFS-6758.02.patch
>
>
> DataXceiver initializes the block size to the default block size for the
> cluster. This size is later used by the FsDatasetImpl when applying
> VolumeChoosingPolicy.
> {code}
> block.setNumBytes(dataXceiverServer.estimateBlockSize);
> {code}
> where
> {code}
> /**
>* We need an estimate for block size to check if the disk partition has
>* enough space. For now we set it to be the default block size set
>* in the server side configuration, which is not ideal because the
>* default block size should be a client-size configuration.
>* A better solution is to include in the header the estimated block size,
>* i.e. either the actual block size or the default block size.
>*/
> final long estimateBlockSize;
> {code}
> In most cases the writer can just pass the maximum expected block size to the
> DN instead of having to use the cluster default.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6878) Change MiniDFSCluster to support StorageType configuration for individual directories
[ https://issues.apache.org/jira/browse/HDFS-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105849#comment-14105849 ] Hudson commented on HDFS-6878: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1870 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1870/]) HDFS-6878. Change MiniDFSCluster to support StorageType configuration for individual directories. (Arpit Agarwal) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619271) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSClusterWithNodeGroup.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestStorageReport.java > Change MiniDFSCluster to support StorageType configuration for individual > directories > - > > Key: HDFS-6878 > URL: https://issues.apache.org/jira/browse/HDFS-6878 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Arpit Agarwal >Priority: Minor > Fix For: 3.0.0, 2.6.0 > > Attachments: HDFS-6878.01.patch, HDFS-6878.02.patch, > HDFS-6878.03.patch > > > Currently, MiniDFSCluster only supports a single StorageType configuration > for all datadnodes, i.e. setting all directories in all datanodes to one > StorageType. It should support setting individual StorageType for each > directory in each datanode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105847#comment-14105847 ] Hudson commented on HDFS-6134: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1870 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1870/]) Fix up CHANGES.txt for HDFS-6134, HADOOP-10150 and related JIRAs. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619203) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt HDFS-6134 and HADOOP-10150 subtasks. Merge fs-encryption branch to trunk. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619197) * /hadoop/common/trunk * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES-fs-encryption.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/config.h.cmake * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/AesCtrCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CipherSuite.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoStreamUtils.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/Decryptor.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/Encryptor.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/JceAesCtrCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslAesCtrCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslCipher.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataOutputStream.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileEncryptionInfo.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/crypto * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/CryptoStreamsTestBase.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoCodec.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/tes
[jira] [Commented] (HDFS-6858) Allow dfs.data.transfer.saslproperties.resolver.class default to hadoop.security.saslproperties.resolver.class
[
https://issues.apache.org/jira/browse/HDFS-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105848#comment-14105848
]
Hudson commented on HDFS-6858:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1870 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1870/])
HDFS-6858. Allow dfs.data.transfer.saslproperties.resolver.class default to
hadoop.security.saslproperties.resolver.class. Contributed by Benoy Antony.
(cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619256)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/DataTransferSaslUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
> Allow dfs.data.transfer.saslproperties.resolver.class default to
> hadoop.security.saslproperties.resolver.class
> --
>
> Key: HDFS-6858
> URL: https://issues.apache.org/jira/browse/HDFS-6858
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.5.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
>Priority: Minor
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6858.patch, HDFS-6858.patch
>
>
> Currently administrator needs to configure both
> {code}dfs.data.transfer.saslproperties.resolver.class{code} and
> {code}hadoop.security.saslproperties.resolver.class{code} to override
> saslproperties for rpc and data transfer properties. In some cases, the
> values for these two properties will be same. In those cases, it may be
> easier if {code}dfs.data.transfer.saslproperties.resolver.class{code}
> defaults to {code}hadoop.security.saslproperties.resolver.class{code}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5135) Umbrella JIRA for NFS end to end unit test frameworks
[ https://issues.apache.org/jira/browse/HDFS-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105866#comment-14105866 ] Brandon Li commented on HDFS-5135: -- The first 4 bytes of the RPC package you got might be the fragment header(RECORD MARKING). On the server side, we use RpcFrameDecoder to process the fragment header (referring bootstrap.setPipelineFactory() in SimpleTcpServer). Let me know if this is not the case. > Umbrella JIRA for NFS end to end unit test frameworks > - > > Key: HDFS-5135 > URL: https://issues.apache.org/jira/browse/HDFS-5135 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.2.0 >Reporter: Brandon Li >Assignee: Zhe Zhang > > Currently, we have to manually start portmap and nfs3 processes to test patch > and new functionalities. This JIRA is to track the effort to introduce a test > framework to NFS unit test without starting standalone nfs3 processes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6899) Allow changing MiniDFSCluster volumes per DN and capacity per volume
[ https://issues.apache.org/jira/browse/HDFS-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6899: Attachment: (was: HDFS-6898-with-6899.02.patch) > Allow changing MiniDFSCluster volumes per DN and capacity per volume > > > Key: HDFS-6899 > URL: https://issues.apache.org/jira/browse/HDFS-6899 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, test >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6899.01.patch, HDFS-6899.02.patch > > > MiniDFSCluster hardcodes the number of directories per volume to two. Propose > removing the hard-coded restriction. > It would be useful to limit the capacity of individual storage directories > for testing purposes. There is already a way to do so for SimulatedFSDataset, > we can add one when using real volumes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6899) Allow changing MiniDFSCluster volumes per DN and capacity per volume
[ https://issues.apache.org/jira/browse/HDFS-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6899: Attachment: (was: HDFS-6899.02.patch) > Allow changing MiniDFSCluster volumes per DN and capacity per volume > > > Key: HDFS-6899 > URL: https://issues.apache.org/jira/browse/HDFS-6899 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, test >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6899.01.patch, HDFS-6899.02.patch > > > MiniDFSCluster hardcodes the number of directories per volume to two. Propose > removing the hard-coded restriction. > It would be useful to limit the capacity of individual storage directories > for testing purposes. There is already a way to do so for SimulatedFSDataset, > we can add one when using real volumes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6899) Allow changing MiniDFSCluster volumes per DN and capacity per volume
[ https://issues.apache.org/jira/browse/HDFS-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6899: Attachment: HDFS-6898-with-6899.02.patch Thanks for looking Jing, added a setter for storageCapacities. > Allow changing MiniDFSCluster volumes per DN and capacity per volume > > > Key: HDFS-6899 > URL: https://issues.apache.org/jira/browse/HDFS-6899 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, test >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6899.01.patch, HDFS-6899.02.patch > > > MiniDFSCluster hardcodes the number of directories per volume to two. Propose > removing the hard-coded restriction. > It would be useful to limit the capacity of individual storage directories > for testing purposes. There is already a way to do so for SimulatedFSDataset, > we can add one when using real volumes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6899) Allow changing MiniDFSCluster volumes per DN and capacity per volume
[ https://issues.apache.org/jira/browse/HDFS-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6899: Attachment: HDFS-6899.02.patch > Allow changing MiniDFSCluster volumes per DN and capacity per volume > > > Key: HDFS-6899 > URL: https://issues.apache.org/jira/browse/HDFS-6899 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, test >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6899.01.patch, HDFS-6899.02.patch > > > MiniDFSCluster hardcodes the number of directories per volume to two. Propose > removing the hard-coded restriction. > It would be useful to limit the capacity of individual storage directories > for testing purposes. There is already a way to do so for SimulatedFSDataset, > we can add one when using real volumes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6899) Allow changing MiniDFSCluster volumes per DN and capacity per volume
[ https://issues.apache.org/jira/browse/HDFS-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6899: Attachment: HDFS-6899.02.patch > Allow changing MiniDFSCluster volumes per DN and capacity per volume > > > Key: HDFS-6899 > URL: https://issues.apache.org/jira/browse/HDFS-6899 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, test >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6899.01.patch, HDFS-6899.02.patch > > > MiniDFSCluster hardcodes the number of directories per volume to two. Propose > removing the hard-coded restriction. > It would be useful to limit the capacity of individual storage directories > for testing purposes. There is already a way to do so for SimulatedFSDataset, > we can add one when using real volumes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6898: Attachment: HDFS-6898.03.patch > DN must reserve space for a full block when an RBW block is created > --- > > Key: HDFS-6898 > URL: https://issues.apache.org/jira/browse/HDFS-6898 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Gopal V >Assignee: Arpit Agarwal > Attachments: HDFS-6898-with-6899.02.patch, HDFS-6898.01.patch, > HDFS-6898.03.patch > > > DN will successfully create two RBW blocks on the same volume even if the > free space is sufficient for just one full block. > One or both block writers may subsequently get a DiskOutOfSpace exception. > This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6898: Status: Open (was: Patch Available) > DN must reserve space for a full block when an RBW block is created > --- > > Key: HDFS-6898 > URL: https://issues.apache.org/jira/browse/HDFS-6898 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Gopal V >Assignee: Arpit Agarwal > Attachments: HDFS-6898-with-6899.02.patch, HDFS-6898.01.patch, > HDFS-6898.03.patch > > > DN will successfully create two RBW blocks on the same volume even if the > free space is sufficient for just one full block. > One or both block writers may subsequently get a DiskOutOfSpace exception. > This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6898: Attachment: (was: HDFS-6898-with-6899.02.patch) > DN must reserve space for a full block when an RBW block is created > --- > > Key: HDFS-6898 > URL: https://issues.apache.org/jira/browse/HDFS-6898 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Gopal V >Assignee: Arpit Agarwal > Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch > > > DN will successfully create two RBW blocks on the same volume even if the > free space is sufficient for just one full block. > One or both block writers may subsequently get a DiskOutOfSpace exception. > This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6851) Flush EncryptionZoneWithId and add an id field to EncryptionZone
[ https://issues.apache.org/jira/browse/HDFS-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6851: Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6891) > Flush EncryptionZoneWithId and add an id field to EncryptionZone > > > Key: HDFS-6851 > URL: https://issues.apache.org/jira/browse/HDFS-6851 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Charles Lamb >Assignee: Charles Lamb > > EncryptionZoneWithId can be flushed by moving the id field up to > EncryptionZone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6581: Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-5851) > Write to single replica in memory > - > > Key: HDFS-6581 > URL: https://issues.apache.org/jira/browse/HDFS-6581 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6581-prototype.01.patch, > HDFSWriteableReplicasInMemory.pdf > > > Per discussion with the community on HDFS-5851, we will implement writing to > a single replica in DN memory via DataTransferProtocol. > This avoids some of the issues with short-circuit writes, which we can > revisit at a later time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6908) incorrect snapshot directory diff generated by snapshot deletion
Juan Yu created HDFS-6908:
-
Summary: incorrect snapshot directory diff generated by snapshot
deletion
Key: HDFS-6908
URL: https://issues.apache.org/jira/browse/HDFS-6908
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Juan Yu
Assignee: Juan Yu
In the following scenario, delete snapshot could generate incorrect snapshot
directory diff and corrupted fsimage, if you restart NN after that, you will
get NullPointerException.
1. create a directory and create a file under it
2. take a snapshot
3. create another file under that directory
4. take second snapshot
5. delete both files and the directory
6. delete second snapshot
incorrect directory diff will be generated.
Restart NN will throw NPE
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.snapshot.FSImageFormatPBSnapshot$Loader.addToDeletedList(FSImageFormatPBSnapshot.java:246)
at
org.apache.hadoop.hdfs.server.namenode.snapshot.FSImageFormatPBSnapshot$Loader.loadDeletedList(FSImageFormatPBSnapshot.java:265)
at
org.apache.hadoop.hdfs.server.namenode.snapshot.FSImageFormatPBSnapshot$Loader.loadDirectoryDiffList(FSImageFormatPBSnapshot.java:328)
at
org.apache.hadoop.hdfs.server.namenode.snapshot.FSImageFormatPBSnapshot$Loader.loadSnapshotDiffSection(FSImageFormatPBSnapshot.java:192)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:254)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:168)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:208)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:906)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:892)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:715)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:653)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:276)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:882)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:629)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:498)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:554)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6851) Flush EncryptionZoneWithId and add an id field to EncryptionZone
[ https://issues.apache.org/jira/browse/HDFS-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105911#comment-14105911 ] Arpit Agarwal commented on HDFS-6851: - Sorry about that, I edited the wrong issue. Restored as sub-task of HDFS-6134. > Flush EncryptionZoneWithId and add an id field to EncryptionZone > > > Key: HDFS-6851 > URL: https://issues.apache.org/jira/browse/HDFS-6851 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Charles Lamb >Assignee: Charles Lamb > > EncryptionZoneWithId can be flushed by moving the id field up to > EncryptionZone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6851) Flush EncryptionZoneWithId and add an id field to EncryptionZone
[ https://issues.apache.org/jira/browse/HDFS-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6851: Issue Type: Sub-task (was: Bug) Parent: HDFS-6134 > Flush EncryptionZoneWithId and add an id field to EncryptionZone > > > Key: HDFS-6851 > URL: https://issues.apache.org/jira/browse/HDFS-6851 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Charles Lamb >Assignee: Charles Lamb > > EncryptionZoneWithId can be flushed by moving the id field up to > EncryptionZone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6909) KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys
Alejandro Abdelnur created HDFS-6909: Summary: KeyProviderCryptoExtension should use CryptoCodec for generation/decryption of keys Key: HDFS-6909 URL: https://issues.apache.org/jira/browse/HDFS-6909 Project: Hadoop HDFS Issue Type: Task Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Currently is using JDK Cipher, with fs-encryption branch merged into trunk we can swap to CryptoCodec. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5851) Support memory as a storage medium
[ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105931#comment-14105931 ] Arpit Agarwal commented on HDFS-5851: - I have just created a branch for HDFS-6581. It is fairly limited in scope compared to the original feature proposal. [~hsaputra], [~shv] the proposal attached to HDFS-6581 should address your queries, please let me know if it doesn't. DDM is not in scope for this phase. > Support memory as a storage medium > -- > > Key: HDFS-5851 > URL: https://issues.apache.org/jira/browse/HDFS-5851 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: > SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, > SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, > SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf > > > Memory can be used as a storage medium for smaller/transient files for fast > write throughput. > More information/design will be added later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5851) Support memory as a storage medium
[ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5851: Component/s: (was: datanode) > Support memory as a storage medium > -- > > Key: HDFS-5851 > URL: https://issues.apache.org/jira/browse/HDFS-5851 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: > SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, > SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, > SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf > > > Memory can be used as a storage medium for smaller/transient files for fast > write throughput. > More information/design will be added later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6910) Initial prototype implementation for replicas in memory using tmpfs
Arpit Agarwal created HDFS-6910: --- Summary: Initial prototype implementation for replicas in memory using tmpfs Key: HDFS-6910 URL: https://issues.apache.org/jira/browse/HDFS-6910 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Arpit Agarwal Assignee: Arpit Agarwal -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6581: Attachment: (was: HDFS-6581-prototype.01.patch) > Write to single replica in memory > - > > Key: HDFS-6581 > URL: https://issues.apache.org/jira/browse/HDFS-6581 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFSWriteableReplicasInMemory.pdf > > > Per discussion with the community on HDFS-5851, we will implement writing to > a single replica in DN memory via DataTransferProtocol. > This avoids some of the issues with short-circuit writes, which we can > revisit at a later time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6910) Initial prototype implementation for replicas in memory using tmpfs
[ https://issues.apache.org/jira/browse/HDFS-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6910: Attachment: HDFS-6910.01.patch > Initial prototype implementation for replicas in memory using tmpfs > --- > > Key: HDFS-6910 > URL: https://issues.apache.org/jira/browse/HDFS-6910 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6910.01.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105954#comment-14105954 ] Arpit Agarwal commented on HDFS-6581: - Moved the patch to sub-task HDFS-6910. > Write to single replica in memory > - > > Key: HDFS-6581 > URL: https://issues.apache.org/jira/browse/HDFS-6581 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFSWriteableReplicasInMemory.pdf > > > Per discussion with the community on HDFS-5851, we will implement writing to > a single replica in DN memory via DataTransferProtocol. > This avoids some of the issues with short-circuit writes, which we can > revisit at a later time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6910) Initial prototype implementation for replicas in memory using tmpfs
[ https://issues.apache.org/jira/browse/HDFS-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6910: Status: Patch Available (was: Open) > Initial prototype implementation for replicas in memory using tmpfs > --- > > Key: HDFS-6910 > URL: https://issues.apache.org/jira/browse/HDFS-6910 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: HDFS-6910.01.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6376) Distcp data between two HA clusters requires another configuration
[
https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105966#comment-14105966
]
Jing Zhao commented on HDFS-6376:
-
Thanks for the response, [~dlmarion].
bq. I have been running a version of this patch for about 2 months on a test
cluster. We are using Hadoop 2 so the patch that I am applying is a little
different.
Cool. Then could you also post an updated patch for hadoop 2 since we will
finally merge the patch to branch-2?
bq. Exclude seemed like a good term for that.
Sounds good to me. Let's keep the current name then.
Some other comments (sorry I should have posted them yesterday...):
# Minor: it may be better to wrap the logic of the following code into a new
method in DFSUtil since we use it in multiple places.
{code}
+Map> newAddressMap =
+DFSUtil.getNNServiceRpcAddresses(conf);
+
+for (String exclude : nameServiceExcludes)
+ newAddressMap.remove(exclude);
{code}
# Currently DFSUtil#getOnlyNameServiceIdOrNull returns null if there are more
than two nameservices specified. There are a couple of places called this
method, and looks like DFSHAAdmin#resolveTarget may hit some issue if no -ns
option is specified by HAAdmin. Thus I think we may also need to add the
exclude logic in DFSUtil#getOnlyNameServiceIdOrNull. And we need to add more
tests for this new feature, e.g., to cover its usage in DFSHAAdmin.
> Distcp data between two HA clusters requires another configuration
> --
>
> Key: HDFS-6376
> URL: https://issues.apache.org/jira/browse/HDFS-6376
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, federation, hdfs-client
>Affects Versions: 2.3.0, 2.4.0
> Environment: Hadoop 2.3.0
>Reporter: Dave Marion
>Assignee: Dave Marion
> Fix For: 3.0.0
>
> Attachments: HDFS-6376-2.patch, HDFS-6376-3-branch-2.4.patch,
> HDFS-6376-4-branch-2.4.patch, HDFS-6376-5-trunk.patch,
> HDFS-6376-6-trunk.patch, HDFS-6376-7-trunk.patch, HDFS-6376-branch-2.4.patch,
> HDFS-6376-patch-1.patch
>
>
> User has to create a third set of configuration files for distcp when
> transferring data between two HA clusters.
> Consider the scenario in [1]. You cannot put all of the required properties
> in core-site.xml and hdfs-site.xml for the client to resolve the location of
> both active namenodes. If you do, then the datanodes from cluster A may join
> cluster B. I can not find a configuration option that tells the datanodes to
> federate blocks for only one of the clusters in the configuration.
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated HDFS-6888: -- Attachment: HDFS-6888-2.patch patch updated. Thank you for your comments. [~kihwal] and [~jira.shegalov]. > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6892) Add XDR packaging method for each NFS request
[ https://issues.apache.org/jira/browse/HDFS-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li reassigned HDFS-6892: Assignee: Brandon Li > Add XDR packaging method for each NFS request > - > > Key: HDFS-6892 > URL: https://issues.apache.org/jira/browse/HDFS-6892 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > > The method can be used for unit tests. > Most request implements this by overriding RequestWithHandle#serialize() > method. However, some request classes missed it, e.g., COMMIT3Request, > MKDIR3Request,READDIR3Request, READDIRPLUS3Request, > RMDIR3RequestREMOVE3Request, SETATTR3Request,SYMLINK3Request. RENAME3Reqeust > is another example. -- This message was sent by Atlassian JIRA (v6.2#6252)
