[jira] [Commented] (HDFS-15253) Set default throttle value on dfs.image.transfer.bandwidthPerSec
[ https://issues.apache.org/jira/browse/HDFS-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081210#comment-17081210 ] Karthik Palanisamy commented on HDFS-15253: --- Thank you [~dineshchitlangia] I think 1M txns even low for normal workload. Most times, I see every 1-2 hours regular checkpoints. I am ok to keep default setting. [~weichiu] what do you think? If we set dfs.image.compress to true then the user can't use dfs.image.parallel.load option. Right now, I will be updating dfs.image.transfer.bandwidthPerSec, and let other settings as it is. > Set default throttle value on dfs.image.transfer.bandwidthPerSec > > > Key: HDFS-15253 > URL: https://issues.apache.org/jira/browse/HDFS-15253 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Major > > The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can > use maximum available bandwidth for fsimage transfers during checkpoint. I > think we should throttle this. Many users were experienced namenode failover > when transferring large image size along with fsimage replication on > dfs.namenode.name.dir. eg. >25Gb. > Thought to set, > dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s) > dfs.namenode.checkpoint.txns=200 (Default is 1M, good to avoid frequent > checkpoint. However, the default checkpoint runs every 6 hours once) > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15271) Update DFSUtil.getSpnegoKeytabKey based on hadoop.http.authentication.kerberos.keytab
Masatake Iwasaki created HDFS-15271:
---
Summary: Update DFSUtil.getSpnegoKeytabKey based on
hadoop.http.authentication.kerberos.keytab
Key: HDFS-15271
URL: https://issues.apache.org/jira/browse/HDFS-15271
Project: Hadoop HDFS
Issue Type: Improvement
Components: security
Affects Versions: 3.3.0
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki
After HADOOP-16314, setting {{dfs.web.authentication.kerberos.keytab}} has no
effect. Keytab file can be specified by
{{hadoop.http.authentication.kerberos.keytab}} now. DFSUtil.getSpnegoKeytabKey
should care about the new proeperty while keeping backward compatibility. Since
the {{getSpnegoKeytabKey}} is only called by NameNodeHttpServer and the value
is not used, deprecating it could be an option.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15217) Add more information to longest write/read lock held log
[
https://issues.apache.org/jira/browse/HDFS-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081309#comment-17081309
]
Toshihiro Suzuki commented on HDFS-15217:
-
I created a PR for this. After this patch, we can see additional information in
the lock report message as follows:
{code:java}
2020-04-11 23:04:36,020 [IPC Server handler 5 on default port 62641] INFO
namenode.FSNamesystem (FSNamesystemLock.java:writeUnlock(321)) - Number of
suppressed write-lock reports: 0
Longest write-lock held at 2020-04-11 23:04:36,020+0900 for 3ms by
delete (ugi=bob (auth:SIMPLE),ip=/127.0.0.1,src=/file,dst=null,perm=null) via
java.lang.Thread.getStackTrace(Thread.java:1559)
org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1058)
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:302)
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:261)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1746)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3274)
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:1130)
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:724)
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:529)
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1016)
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:944)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2948)
Total suppressed write-lock held time: 0.0
{code}
This patch adds the additional information *"by delete (ugi=bob
(auth:SIMPLE),ip=/127.0.0.1,src=/file,dst=null,perm=null)"* which is similar to
the audit log format.
> Add more information to longest write/read lock held log
>
>
> Key: HDFS-15217
> URL: https://issues.apache.org/jira/browse/HDFS-15217
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
>
> Currently, we can see the stack trace in the longest write/read lock held
> log, but sometimes we need more information, for example, a target path of
> deletion:
> {code:java}
> 2020-03-10 21:51:21,116 [main] INFO namenode.FSNamesystem
> (FSNamesystemLock.java:writeUnlock(276)) - Number of suppressed
> write-lock reports: 0
> Longest write-lock held at 2020-03-10 21:51:21,107+0900 for 6ms via
> java.lang.Thread.getStackTrace(Thread.java:1559)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1058)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:257)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:233)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1706)
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3188)
> ...
> {code}
> Adding more information (opName, path, etc.) to the log is useful to
> troubleshoot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15243) Child directory should not be deleted or renamed if parent directory is a protected directory
[ https://issues.apache.org/jira/browse/HDFS-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081344#comment-17081344 ] Ayush Saxena commented on HDFS-15243: - Thanx [~rain_lyy] for the patch. * Please include a UT covering all the changes * Move the configuration to HDFS rather than being at common, since this would be only HDFS specific > Child directory should not be deleted or renamed if parent directory is a > protected directory > - > > Key: HDFS-15243 > URL: https://issues.apache.org/jira/browse/HDFS-15243 > Project: Hadoop HDFS > Issue Type: Bug > Components: 3.1.1 >Affects Versions: 3.1.1 >Reporter: liuyanyu >Assignee: liuyanyu >Priority: Major > Attachments: HDFS-15243.001.patch, image-2020-03-28-09-23-31-335.png > > > HDFS-8983 add fs.protected.directories to support protected directories on > NameNode. But as I test, when set a parent directory（eg /testA) to > protected directory, the child directory (eg /testA/testB) still can be > deleted or renamed. When we protect a directory mainly for protecting the > data under this directory , So I think the child directory should not be > delete or renamed if the parent directory is a protected directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11041) Unable to unregister FsDatasetState MBean if if DataNode is shutdown twice
[
https://issues.apache.org/jira/browse/HDFS-11041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081367#comment-17081367
]
Ayush Saxena commented on HDFS-11041:
-
Thanx [~weichiu] for the report.
Is this issue still there? Any reasons for holding this?
> Unable to unregister FsDatasetState MBean if if DataNode is shutdown twice
> --
>
> Key: HDFS-11041
> URL: https://issues.apache.org/jira/browse/HDFS-11041
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Trivial
> Attachments: HDFS-11041.01.patch, HDFS-11041.02.patch,
> HDFS-11041.03.patch
>
>
> I saw error message like the following in some tests
> {noformat}
> 2016-10-21 04:09:03,900 [main] WARN util.MBeans
> (MBeans.java:unregister(114)) - Error unregistering
> Hadoop:service=DataNode,name=FSDatasetState-33cd714c-0b1a-471f-8efe-f431d7d874bc
> javax.management.InstanceNotFoundException:
> Hadoop:service=DataNode,name=FSDatasetState-33cd714c-0b1a-471f-8efe-f431d7d874bc
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
> at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:112)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdown(FsDatasetImpl.java:2127)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:2016)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1985)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1962)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1936)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1929)
> at
> org.apache.hadoop.hdfs.TestDatanodeReport.testDatanodeReport(TestDatanodeReport.java:144)
> {noformat}
> The test shuts down datanode, and then shutdown cluster, which shuts down the
> a datanode twice. Resetting the FsDatasetSpi reference in DataNode to null
> resolves the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15247) RBF: Provide Non DFS Used per DataNode in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-15247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-15247: --- Attachment: HDFS-15247.002.patch > RBF: Provide Non DFS Used per DataNode in DataNode UI > - > > Key: HDFS-15247 > URL: https://issues.apache.org/jira/browse/HDFS-15247 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-15247.001.patch, HDFS-15247.002.patch, Screenshot > from 2020-04-09 17-19-30.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15247) RBF: Provide Non DFS Used per DataNode in DataNode UI
[ https://issues.apache.org/jira/browse/HDFS-15247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-15247: --- Attachment: (was: HDFS-15247.002.patch) > RBF: Provide Non DFS Used per DataNode in DataNode UI > - > > Key: HDFS-15247 > URL: https://issues.apache.org/jira/browse/HDFS-15247 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-15247.001.patch, Screenshot from 2020-04-09 > 17-19-30.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12862) CacheDirective becomes invalid when NN restart or failover
[
https://issues.apache.org/jira/browse/HDFS-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12862:
Target Version/s: 3.4.0
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> CacheDirective becomes invalid when NN restart or failover
> --
>
> Key: HDFS-12862
> URL: https://issues.apache.org/jira/browse/HDFS-12862
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching, hdfs
>Affects Versions: 2.7.1
> Environment:
>Reporter: Wang XL
>Assignee: Wang XL
>Priority: Major
> Labels: patch
> Fix For: 3.3.0
>
> Attachments: HDFS-12862-branch-2.7.1.001.patch,
> HDFS-12862-trunk.002.patch, HDFS-12862-trunk.003.patch,
> HDFS-12862-trunk.004.patch, HDFS-12862.005.patch, HDFS-12862.006.patch,
> HDFS-12862.007.patch, HDFS-12862.branch-3.1.patch
>
>
> The logic in FSNDNCacheOp#modifyCacheDirective is not correct. when modify
> cacheDirective,the expiration in directive may be a relative expiryTime, and
> EditLog will serial a relative expiry time.
> {code:java}
> // Some comments here
> static void modifyCacheDirective(
> FSNamesystem fsn, CacheManager cacheManager, CacheDirectiveInfo
> directive,
> EnumSet flags, boolean logRetryCache) throws IOException {
> final FSPermissionChecker pc = getFsPermissionChecker(fsn);
> cacheManager.modifyDirective(directive, pc, flags);
> fsn.getEditLog().logModifyCacheDirectiveInfo(directive, logRetryCache);
> }
> {code}
> But when SBN replay the log ,it will invoke
> FSImageSerialization#readCacheDirectiveInfo as a absolute expiryTime.It will
> result in the inconsistency .
> {code:java}
> public static CacheDirectiveInfo readCacheDirectiveInfo(DataInput in)
> throws IOException {
> CacheDirectiveInfo.Builder builder =
> new CacheDirectiveInfo.Builder();
> builder.setId(readLong(in));
> int flags = in.readInt();
> if ((flags & 0x1) != 0) {
> builder.setPath(new Path(readString(in)));
> }
> if ((flags & 0x2) != 0) {
> builder.setReplication(readShort(in));
> }
> if ((flags & 0x4) != 0) {
> builder.setPool(readString(in));
> }
> if ((flags & 0x8) != 0) {
> builder.setExpiration(
> CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)));
> }
> if ((flags & ~0xF) != 0) {
> throw new IOException("unknown flags set in " +
> "ModifyCacheDirectiveInfoOp: " + flags);
> }
> return builder.build();
> }
> {code}
> In other words， fsn.getEditLog().logModifyCacheDirectiveInfo(directive,
> logRetryCache) may serial a relative expiry time,But
> builder.setExpiration(CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)))
>read it as a absolute expiryTime.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10919) Provide admin/debug tool to dump out info of a given block
[ https://issues.apache.org/jira/browse/HDFS-10919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10919: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Provide admin/debug tool to dump out info of a given block > -- > > Key: HDFS-10919 > URL: https://issues.apache.org/jira/browse/HDFS-10919 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Reporter: Yongjun Zhang >Assignee: Shweta >Priority: Major > > We have fsck to find out blocks associated with a file, which is nice. > Sometimes, we saw trouble with a specific block, we'd like to collect info > about this block, such as > * what file this block belong to, > * where the replicas of this block are located, > * whether the block is EC coded; > * if a block is EC coded, whether it's a data block, or code > * if a block is EC coded, what's the codec. > * if a block is EC coded, what's the block group > * for the block group, what are the other blocks > Create this jira to provide such a util, as dfsadmin, or a debug tool. > Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11942) make new chooseDataNode policy work in more operation like seek, fetch
[ https://issues.apache.org/jira/browse/HDFS-11942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11942: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > make new chooseDataNode policy work in more operation like seek, fetch > > > Key: HDFS-11942 > URL: https://issues.apache.org/jira/browse/HDFS-11942 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: 2.6.0, 2.7.0, 3.0.0-alpha3 >Reporter: Fangyuan Deng >Priority: Major > Attachments: HDFS-11942.0.patch, HDFS-11942.1.patch, > ssd-first-disable(default).png, ssd-first-enable.png > > > in default policy, if a file is ONE_SSD, client will prior read the local > disk replica rather than the remote ssd replica. > but now, the pci-e SSD and 10G ethernet make remote read SSD more faster than > the local disk. > HDFS-9666 give us a patch, but the code is not complete and not updated for > a long time. > this sub-task issue give a complete patch and > we have tested on three machines [ 32 core cpu, 128G mem , 1000M network, > 1.2T HDD, 800G SSD(intel P3600) ]. > with this feather, throughput of hbase table(ONE_SSD) is double of which > without this feather -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11161) Incorporate Baidu Yun BOS file system implementation
[ https://issues.apache.org/jira/browse/HDFS-11161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11161: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Incorporate Baidu Yun BOS file system implementation > > > Key: HDFS-11161 > URL: https://issues.apache.org/jira/browse/HDFS-11161 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Reporter: Faen Zhang >Priority: Major > Original Estimate: 840h > Remaining Estimate: 840h > > Baidu Yun ( https://cloud.baidu.com/ ) is one of top tier cloud computing > provider. Baidu Yun BOS is widely used among China's cloud users, but > currently it is not easy to access data laid on BOS storage from user's > Hadoop/Spark application, because of no original support for BOS in Hadoop. > This work aims to integrate Baidu Yun BOS with Hadoop. By simple > configuration, Spark/Hadoop applications can read/write data from BOS without > any code change. Narrowing the gap between user's APP and data storage, like > what have been done for S3 and Aliyun OSS in Hadoop. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15131) FoldedTreeSet appears to degrade over time
[
https://issues.apache.org/jira/browse/HDFS-15131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-15131:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> FoldedTreeSet appears to degrade over time
> --
>
> Key: HDFS-15131
> URL: https://issues.apache.org/jira/browse/HDFS-15131
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, namenode
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
>
> We have seen some occurrences of the Namenode getting very slow on delete
> operations, to the point where IBRs get blocked frequently and files fail to
> close. On one cluster in particular, after about 4 weeks of uptime, the
> Namenode started responding very poorly. Restarting it corrected the problem
> for another 4 weeks.
> In that example, jstacks in the namenode always pointed to slow operations
> around a HDFS delete call which was performing an operation on the
> FoldedTreeSet structure. The captured jstacks always pointed at an operation
> on the folded tree set each time they were sampled:
> {code}
> "IPC Server handler 573 on 8020" #663 daemon prio=5 os_prio=0
> tid=0x7fe6a4087800 nid=0x97a6 runnable [0x7fe67bdfd000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.removeAndGet(FoldedTreeSet.java:879)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.remove(FoldedTreeSet.java:911)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.removeBlock(DatanodeStorageInfo.java:263)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:194)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:108)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlockFromMap(BlockManager.java:3676)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlock(BlockManager.java:3507)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeBlocks(FSNamesystem.java:4158)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:4132)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:4069)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:4053)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:845)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.delete(AuthorizationProviderProxyClientProtocol.java:308)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:603)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> {code}
> The observation in this case, was that the namenode worked fine after a
> restart and then at some point after about 4 weeks of uptime, this problem
> started happening, and it would persist until the namenode was restarted.
> Then the problem did not return for about another 4 weeks.
> On a completely different cluster and version, I recently came across a
> problem where files were again failing to close (last block does not have
> sufficient number of replicas) and the datanodes were logging a lot of
> messages like the following:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> These messages had a range of durations and were fairly frequent. Focusing on
> the longer messages at around 20 seconds and checking a few different
> occurrences, jstacks showed as very similar problem to the NN issue, in that
> the folderTreeSet seems to be at the root of the problem.
> The heartbeat thread is waiting on a lock:
> {code}
> "Thread-26" #243 daemon prio=5 os_prio=0 tid=0x7f575225a000 nid=0x1d8bae
> waiting for monitor entry [0x7f571ed76000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.ap
[jira] [Updated] (HDFS-8430) Erasure coding: compute file checksum for striped files (stripe by stripe)
[
https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-8430:
---
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Erasure coding: compute file checksum for striped files (stripe by stripe)
> --
>
> Key: HDFS-8430
> URL: https://issues.apache.org/jira/browse/HDFS-8430
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Walter Su
>Assignee: Kai Zheng
>Priority: Major
> Attachments: HDFS-8430-poc1.patch
>
>
> HADOOP-3981 introduces a distributed file checksum algorithm. It's designed
> for replicated block.
> {{DFSClient.getFileChecksum()}} need some updates, so it can work for striped
> block group.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13287) TestINodeFile#testGetBlockType results in NPE when run alone
[
https://issues.apache.org/jira/browse/HDFS-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13287:
Target Version/s: 3.4.0
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> TestINodeFile#testGetBlockType results in NPE when run alone
>
>
> Key: HDFS-13287
> URL: https://issues.apache.org/jira/browse/HDFS-13287
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Minor
> Fix For: 3.3.0, 3.1.4
>
> Attachments: HDFS-13287.01.patch
>
>
> When TestINodeFile#testGetBlockType is run by itself, it results in the
> following error:
> {code:java}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.218
> s <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.TestINodeFile
> [ERROR]
> testGetBlockType(org.apache.hadoop.hdfs.server.namenode.TestINodeFile) Time
> elapsed: 0.023 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getPolicyInfoByID(ErasureCodingPolicyManager.java:220)
> at
> org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.getByID(ErasureCodingPolicyManager.java:208)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(INodeFile.java:207)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.(INodeFile.java:266)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.createStripedINodeFile(TestINodeFile.java:112)
> at
> org.apache.hadoop.hdfs.server.namenode.TestINodeFile.testGetBlockType(TestINodeFile.java:299)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13998) ECAdmin NPE with -setPolicy -replicate
[
https://issues.apache.org/jira/browse/HDFS-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13998:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> ECAdmin NPE with -setPolicy -replicate
> --
>
> Key: HDFS-13998
> URL: https://issues.apache.org/jira/browse/HDFS-13998
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Xiao Chen
>Assignee: Zsolt Venczel
>Priority: Major
> Attachments: HDFS-13998.01.patch, HDFS-13998.02.patch,
> HDFS-13998.03.patch
>
>
> HDFS-13732 tried to improve the output of the console tool. But we missed the
> fact that for replication, {{getErasureCodingPolicy}} would return null.
> This jira is to fix it in ECAdmin, and add a unit test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11503) Integrate Chocolate Cloud RS coder implementation
[ https://issues.apache.org/jira/browse/HDFS-11503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11503: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Integrate Chocolate Cloud RS coder implementation > - > > Key: HDFS-11503 > URL: https://issues.apache.org/jira/browse/HDFS-11503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: Andrew Wang >Assignee: Marcell Feher >Priority: Major > Attachments: HDFS-11503.patch > > > Quote from Marcell on HDFS-7285: > First of all let me introduce ourselves: we are Chocolate Cloud from Denmark, > we use erasure coding to improve storage solutions. We already have > Reed-Solomon and Random Linear Network Coding backends for Liberasurecode, > and now we are at the final stage of developing our RS plugin to HDFS-EC. The > performance of our plugin is similar to ISA-L's, in some configurations we > are better, in others we are worse (our initial speed comparison charts can > be found here: https://www.chocolate-cloud.cc/Plugins/HDFS-EC/hdfs.html). > We would like our plugin to become officially supported in Hadoop 3.0. We can > already provide a preliminary version of our (native) library and a patch > with the necessary glue code for the next alpha release. > I'd like to know your thoughts about whether it's possible and how it could > be achieved. > P.S: I'm happy to share more details if there's interest -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14903) Update access time in toCompleteFile
[ https://issues.apache.org/jira/browse/HDFS-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14903: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Update access time in toCompleteFile > > > Key: HDFS-14903 > URL: https://issues.apache.org/jira/browse/HDFS-14903 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: lihanran >Assignee: lihanran >Priority: Major > Attachments: HDFS-14903.001.patch > > > when cleate a file,accesstime and modifitime are different -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10274) Move NameSystem#isInStartupSafeMode() to BlockManagerSafeMode
[
https://issues.apache.org/jira/browse/HDFS-10274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-10274:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Move NameSystem#isInStartupSafeMode() to BlockManagerSafeMode
> -
>
> Key: HDFS-10274
> URL: https://issues.apache.org/jira/browse/HDFS-10274
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Attachments: HDFS-10274-01.patch
>
>
> To reduce the number of methods in Namesystem interface and for clean looking
> refactor, its better to move {{isInStartupSafeMode()}} to BlockManager and
> BlockManagerSafeMode, as most of the callers are in BlockManager. So one more
> interface overhead can be reduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7273) Add start and stop wrapper scripts for mover
[ https://issues.apache.org/jira/browse/HDFS-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-7273: --- Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Add start and stop wrapper scripts for mover > > > Key: HDFS-7273 > URL: https://issues.apache.org/jira/browse/HDFS-7273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > > Similar to balancer , we need start/stop mover scripts to run data migration > tool as a daemon. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11260) Slow writer threads are not stopped
[
https://issues.apache.org/jira/browse/HDFS-11260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11260:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Slow writer threads are not stopped
> ---
>
> Key: HDFS-11260
> URL: https://issues.apache.org/jira/browse/HDFS-11260
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.0
> Environment: CDH5.8.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
>
> If a DataNode receives a transferred block, it tries to stop writer to the
> same block. However, this may not work, and we saw the following error
> message and stacktrace.
> Fundamentally, the assumption of {{ReplicaInPipeline#stopWriter}} is wrong.
> It assumes the writer thread must be a DataXceiver thread, which it can be
> interrupted and terminates afterwards. However, IPC threads may also be the
> writer thread by calling initReplicaRecovery, and which ignores interrupt and
> do not terminate.
> {noformat}
> 2016-12-16 19:58:56,167 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Join on writer thread Thread[IPC Server handler 6 on 50020,5,main] timed out
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
> org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:135)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2052)
> 2016-12-16 19:58:56,167 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver constructor. Cause is
> 2016-12-16 19:58:56,168 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> sj1dra082.corp.adobe.com:50010:DataXceiver error processing WRITE_BLOCK
> operation src: /10.10.0.80:44105 dst: /10.10.0.82:50010
> java.io.IOException: Join on writer thread Thread[IPC Server handler 6 on
> 50020,5,main] timed out
> at
> org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.stopWriter(ReplicaInPipeline.java:212)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:1579)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:195)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:669)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:169)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:106)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> There is also a logic error in FsDatasetImpl#createTemporary, in which if the
> code in the synchronized block executes for more than 60 seconds (in theory),
> it could throw an exception, without trying to stop the existing slow writer.
> We saw a FsDatasetImpl#createTemporary failed after nearly 10 minutes, and
> it's unclear why yet. It's my understanding that the code intends to stop
> slow writers after 1 minute by default. Some code rewrite is probably needed
> to get the logic right.
> {noformat}
> 2016-12-16 23:12:24,636 WARN
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Unable
> to stop existing writer for block
> BP-1527842723-10.0.0.180-1367984731269:blk_4313782210_1103780331023 after
> 568320 miniseconds.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14353: Target Version/s: 3.4.0 Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch, > HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch, > HDFS-14353.006.patch, HDFS-14353.007.patch, HDFS-14353.008.patch, > HDFS-14353.009.patch, screenshot-1.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14897) [hadoop-hdfs] Fix order of actual and expected expression in assert statements
[
https://issues.apache.org/jira/browse/HDFS-14897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-14897:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> [hadoop-hdfs] Fix order of actual and expected expression in assert statements
> --
>
> Key: HDFS-14897
> URL: https://issues.apache.org/jira/browse/HDFS-14897
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> Fix order of actual and expected expression in assert statements which gives
> misleading message when test case fails. Attached file has some of the places
> where it is placed wrongly.
> {code:java}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> {code}
> For long term, [AssertJ|http://joel-costigliola.github.io/assertj/] can be
> used for new test cases which avoids such mistakes.
> This is a follow-up jira for the hadoop-hdfs project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14261) Kerberize JournalNodeSyncer unit test
[ https://issues.apache.org/jira/browse/HDFS-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14261: Target Version/s: 3.4.0 Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Kerberize JournalNodeSyncer unit test > - > > Key: HDFS-14261 > URL: https://issues.apache.org/jira/browse/HDFS-14261 > Project: Hadoop HDFS > Issue Type: Test > Components: journal-node, security, test >Affects Versions: 3.2.0, 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14261.001.patch > > > This jira is an addition to HDFS-14140. Making the unit tests in > TestJournalNodeSync run on a Kerberized cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12208) NN should consider DataNode#xmitInProgress when placing new block
[ https://issues.apache.org/jira/browse/HDFS-12208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12208: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > NN should consider DataNode#xmitInProgress when placing new block > - > > Key: HDFS-12208 > URL: https://issues.apache.org/jira/browse/HDFS-12208 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, erasure-coding >Affects Versions: 3.0.0-alpha4 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu >Priority: Minor > > As discussed in HDFS-12044, NN only considers xceiver counts on DN when > placing new blocks. NN should also consider background reconstruction works, > presented by xmits on DN. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14904) Balancer should pick nodes based on utilization in each iteration
[ https://issues.apache.org/jira/browse/HDFS-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14904: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Balancer should pick nodes based on utilization in each iteration > - > > Key: HDFS-14904 > URL: https://issues.apache.org/jira/browse/HDFS-14904 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Major > > In each iteration, balancer should pick nodes with the highest/lowest usage > first, as suggested in HDFS-14894 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14607) Support OpenTracing in WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14607: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Support OpenTracing in WebHDFS > -- > > Key: HDFS-14607 > URL: https://issues.apache.org/jira/browse/HDFS-14607 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Nikhil Navadiya >Assignee: Nikhil Navadiya >Priority: Major > Attachments: HDFS-14607.WIP.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11611) Handle unsuccessful encode and decode calls
[ https://issues.apache.org/jira/browse/HDFS-11611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11611: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Handle unsuccessful encode and decode calls > --- > > Key: HDFS-11611 > URL: https://issues.apache.org/jira/browse/HDFS-11611 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: László Bence Nagy >Priority: Minor > > Normally encode and decode operations are performed successfully but there > can be some rare cases when they fail for some reason, specifically with the > native implementations. The framework should be ready to perceive these cases > (for example by catching exceptions or by checking the operations return > value) and then do some kind of rescheduling of the failed operation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11927) hdfs oiv_legacy:Normalize the verification of input parameter
[ https://issues.apache.org/jira/browse/HDFS-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11927: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > hdfs oiv_legacy:Normalize the verification of input parameter > - > > Key: HDFS-11927 > URL: https://issues.apache.org/jira/browse/HDFS-11927 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0-alpha3 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Attachments: HDFS-11927.002.patch, HDFS-11927.patch > > > At present, hdfs oiv_legacy tool lacks in verification of input > parameter.people can type in irrelevant option like: > bq../hdfs oiv_legacy -i fsimage_000 -o out -p XML -step 1024 > or type in option with wrong format which he think come into effec but > actually not: > bq../hdfs oiv_legacy -i fsimage_000 -o out -p > FileDistribution maxSize 4096 step 512 format > or some meaningless word which can also get through: > bq../hdfs oiv_legacy -i fsimage_000 -o out -p XML Hello World > We'd better not let these cases go unchecked. > By the way, this patch fixed some checkstyle warnings in > +DelimitedImageVisitor.java+ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12332) Logging improvement suggestion for SampleStat function MinMax.add
[
https://issues.apache.org/jira/browse/HDFS-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12332:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Logging improvement suggestion for SampleStat function MinMax.add
> -
>
> Key: HDFS-12332
> URL: https://issues.apache.org/jira/browse/HDFS-12332
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: metrics
>Affects Versions: 3.0.0-alpha4
>Reporter: Xu Zhao
>Priority: Minor
> Attachments: hdfs-12332.patch
>
>
> In order to debug performance anomlies, it is better to see when the max and
> min value get updated in metrics.
> For example, in our workload we would like to see the longest latency of a
> write block request on DataNode.
> The following metric updating code could possibly update the min/max value in
> SampleStat class:
> {code:java}
> datanode.getMetrics().addWriteBlockOp(elapsed());
> datanode.getMetrics().incrWritesFromClient(peer.isLocal(), size);
> {code}
> Here I am attaching an attempt to patch it.
> The patch just adds additional DEBUG level logging in the MinMax class and
> prints the new value when it gets updated.
> Please let me know what you think. For example, if this patch would introduce
> too much noises, what is the best way to trace the read/writeBlock that has
> the minimum/maximum latency?
> Any comments are appreciated!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11020) Add more doc for HDFS transparent encryption
[ https://issues.apache.org/jira/browse/HDFS-11020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11020: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Add more doc for HDFS transparent encryption > > > Key: HDFS-11020 > URL: https://issues.apache.org/jira/browse/HDFS-11020 > Project: Hadoop HDFS > Issue Type: Task > Components: documentation, encryption, fs >Reporter: Yi Liu >Assignee: Jennica Pounds >Priority: Minor > > We need correct version of Openssl which supports hardware acceleration of > AES CTR. > Let's add more doc about how to configure the correct Openssl. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11542) Fix RawErasureCoderBenchmark decoding operation
[ https://issues.apache.org/jira/browse/HDFS-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11542: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Fix RawErasureCoderBenchmark decoding operation > --- > > Key: HDFS-11542 > URL: https://issues.apache.org/jira/browse/HDFS-11542 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: László Bence Nagy >Priority: Minor > Labels: test > > There are some issues with the decode operation in the > *RawErasureCoderBenchmark.java* file. The decoding method is called like > this: *decoder.decode(decodeInputs, ERASED_INDEXES, outputs);*. > Using RS 6+3 configuration it could be called with these parameters correctly > like this: *decode([ d0, NULL, d2, d3, NULL, d5, p0, NULL, p2 ], [ 1, 4, 7 ], > [ -, -, - ])*. The 1,4,7 indexes are in the *ERASED_INDEXES* array so in the > *decodeInputs* array the values at those indexes are set to NULL, all other > data and parity packets are present in the array. The *outputs* array's > length is 3, where the d1, d4 and p1 packets should be reconstructed. This > would be the right solution. > Right now this example would be called like this: *decode([ d0, d1, d2, d3, > d4, d5, -, -, - ], [ 1, 4, 7 ], [ -, -, - ])*. So it has two main problems > with the *decodeInputs* array. Firstly, the packets are not set to NULL where > they should be based on the *ERASED_INDEXES* array. Secondly, it does not > have any parity packets for decoding. > The first problem is easy to solve, the values at the proper indexes need to > be set to NULL. The latter one is a little harder because right now multiple > rounds of encode operations are done one after another and similarly multiple > decode operations are called one by one. Encode and decode pairs should be > called one after another so that the encoded parity packets can be used in > the *decodeInputs* array as a parameter for decode. (Of course, their > performance should be still measured separately.) > Moreover, there is one more problem in this file. Right now it works with RS > 6+3 and the *ERASED_INDEXES* array is fixed to *[ 6, 7, 8 ]*. So the three > parity packets are needed to be reconstructed. This means that no real decode > performance is measured because no data packet is needed to be reconstructed > (even if the decode works properly). Actually, only new parity packets are > needed to be encoded. The exact implementation depends on the underlying > erasure coding plugin, but the point is that data packets should also be > erased to measure real decode performance. > In addition to this, more RS configurations (not just 6+3) could be measured > as well to be able to compare them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13064) Httpfs should return json instead of html when writting to a file without Content-Type
[
https://issues.apache.org/jira/browse/HDFS-13064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13064:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Httpfs should return json instead of html when writting to a file without
> Content-Type
> --
>
> Key: HDFS-13064
> URL: https://issues.apache.org/jira/browse/HDFS-13064
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Affects Versions: 3.0.0
>Reporter: fang zhenyi
>Assignee: fang zhenyi
>Priority: Minor
> Attachments: HDFS-13064.001.patch, HDFS-13064.002.patch
>
>
> When I create a hdfs file, I get the following response.
>
> {code:java}
> zdh102:~ # curl -i -X PUT
> "http://10.43.183.103:14000/webhdfs/v1/2.txt?op=CREATE&user.name=hdfs&data=true";
> HTTP/1.1 400 Bad Request
> Server: Apache-Coyote/1.1
> Set-Cookie:
> hadoop.auth="u=hdfs&p=hdfs&t=simple&e=1516901333684&s=wYqDlu/ovRxay9d6I6UmoH77KKI=";
> Path=/; Expires= , 25- -2018 17:28:53 GMT; HttpOnly
> Content-Type: text/html;charset=utf-8
> Content-Language: en
> Content-Length: 1122
> Date: Thu, 25 Jan 2018 07:28:53 GMT
> Connection: close
> Apache Tomcat/7.0.82 - Error report
> HTTP Status 400 - Data upload requests must have
> content-type set to 'application/octet-stream' noshade="noshade">type Status reportmessage Data
> upload requests must have content-type set to
> 'application/octet-stream'description The request sent
> by the client was syntactically incorrect. noshade="noshade">Apache Tomcat/7.0.82zdh102:~ #
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12261) Improve HdfsAdmin's javadoc regarding superusers.
[
https://issues.apache.org/jira/browse/HDFS-12261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12261:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Improve HdfsAdmin's javadoc regarding superusers.
> -
>
> Key: HDFS-12261
> URL: https://issues.apache.org/jira/browse/HDFS-12261
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
>
> From [discussions
> of|https://issues.apache.org/jira/browse/HDFS-10899?focusedCommentId=16113267&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16113267]
> HDFS-10899, we should improve the javadoc accordingly.
> {quote}
> This is not actually true, this class is just for HDFS-specific operations.
> Putting "Admin" in the name is a misnomer, and since this continues to be
> confusing, maybe we should enhance the class javadoc to make this explicit.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12084) Scheduled Count will not decrement when file is deleted before all IBR's received
[ https://issues.apache.org/jira/browse/HDFS-12084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12084: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Scheduled Count will not decrement when file is deleted before all IBR's > received > - > > Key: HDFS-12084 > URL: https://issues.apache.org/jira/browse/HDFS-12084 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12084-001.patch, HDFS-12084-002.patch, > HDFS-12084-003.patch, HDFS-12084-branch-2.patch > > > When small files creation && deletion happens so frequently and DN's did not > report blocks to NN before deletion, then scheduled count will keep on > increment and which will not deleted as blocks are deleted. > *Note*: Every 20 mins,this can be rolled, but with in 20 mins, count can be > more as so many operations. > when batchIBR enabled with committed allowed=1 this will be observed more. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11397) TestThrottledAsyncChecker#testCancellation timed out
[
https://issues.apache.org/jira/browse/HDFS-11397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11397:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> TestThrottledAsyncChecker#testCancellation timed out
>
>
> Key: HDFS-11397
> URL: https://issues.apache.org/jira/browse/HDFS-11397
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, test
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Assignee: Manjunath Anand
>Priority: Minor
> Attachments: HDFS-11397-V01.patch
>
>
> {noformat}
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 61.153 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker
> testCancellation(org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker)
> Time elapsed: 60.033 sec <<< ERROR!
> java.lang.Exception: test timed out after 6 milliseconds
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker.testCancellation(TestThrottledAsyncChecker.java:114)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12763) DataStreamer should heartbeat during flush
[ https://issues.apache.org/jira/browse/HDFS-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12763: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > DataStreamer should heartbeat during flush > -- > > Key: HDFS-12763 > URL: https://issues.apache.org/jira/browse/HDFS-12763 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.1 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: HDFS-12763.001.patch > > > From HDFS-5032: > bq. Absence of heartbeat during flush will be fixed in a separate jira by > Daryn Sharp > This JIRA tracks the case where absence of heartbeat can cause the pipeline > to fail if operations like flush take some time to complete. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11398) TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure still fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-11398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11398:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure still fails
> intermittently
>
>
> Key: HDFS-11398
> URL: https://issues.apache.org/jira/browse/HDFS-11398
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-11398-reproduce.patch, HDFS-11398.001.patch,
> HDFS-11398.002.patch, failure.log
>
>
> The test {{TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure}}
> still fails intermittently in trunk after HDFS-11316. The stack infos:
> {code}
> testUnderReplicationAfterVolFailure(org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure)
> Time elapsed: 95.021 sec <<< ERROR!
> java.util.concurrent.TimeoutException: Timed out waiting for condition.
> Thread diagnostics:
> Timestamp: 2017-02-07 07:00:34,193
>
> java.lang.Thread.State: RUNNABLE
> at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native
> Method)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52)
> at
> org.apache.hadoop.net.unix.DomainSocketWatcher$2.run(DomainSocketWatcher.java:511)
> at java.lang.Thread.run(Thread.java:745)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:276)
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure.testUnderReplicationAfterVolFailure(TestDataNodeVolumeFailure.java:412)
> {code}
> I looked into this and found there is one chance that the vaule
> {{UnderReplicatedBlocksCount}} will be no longer > 0. The following is my
> analysation:
> In test {{TestDataNodeVolumeFailure.testUnderReplicationAfterVolFailure}}, it
> uses creating file to trigger the disk error checking. The related codes:
> {code}
> Path file1 = new Path("/test1");
> DFSTestUtil.createFile(fs, file1, 1024, (short)3, 1L);
> DFSTestUtil.waitReplication(fs, file1, (short)3);
> // Fail the first volume on both datanodes
> File dn1Vol1 = new File(dataDir, "data"+(2*0+1));
> File dn2Vol1 = new File(dataDir, "data"+(2*1+1));
> DataNodeTestUtils.injectDataDirFailure(dn1Vol1, dn2Vol1);
> Path file2 = new Path("/test2");
> DFSTestUtil.createFile(fs, file2, 1024, (short)3, 1L);
> DFSTestUtil.waitReplication(fs, file2, (short)3);
> {code}
> This will lead one problem: If the cluster is busy, and it costs long time to
> wait replication of file2 to be desired value. During this time, the under
> replication blocks of file1 can also be rereplication in cluster. If this is
> done, the condition {{underReplicatedBlocks > 0}} will never be satisfied.
> And this can be reproduced in my local env.
> Actually, we can use a easy way {{DataNodeTestUtils.waitForDiskError}} to
> replace this, it runs fast and be more reliable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13639) SlotReleaser is not fast enough
[
https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13639:
Target Version/s: 3.4.0
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> SlotReleaser is not fast enough
> ---
>
> Key: HDFS-13639
> URL: https://issues.apache.org/jira/browse/HDFS-13639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.4.0, 2.6.0, 3.0.2
> Environment: 1. YCSB:
> {color:#00} recordcount=20
> fieldcount=1
> fieldlength=1000
> operationcount=1000
>
> workload=com.yahoo.ycsb.workloads.CoreWorkload
>
> table=ycsb-test
> columnfamily=C
> readproportion=1
> updateproportion=0
> insertproportion=0
> scanproportion=0
>
> maxscanlength=0
> requestdistribution=zipfian
>
> # default
> readallfields=true
> writeallfields=true
> scanlengthdistribution=constan{color}
> {color:#00}2. datanode:{color}
> -Xmx2048m -Xms2048m -Xmn1024m -XX:MaxDirectMemorySize=1024m
> -XX:MaxPermSize=256m -Xloggc:$run_dir/stdout/datanode_gc_${start_time}.log
> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=$log_dir -XX:+PrintGCApplicationStoppedTime
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
> -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=1
> -XX:+CMSScavengeBeforeRemark -XX:+PrintPromotionFailure
> -XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -verbose:gc -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps
> {color:#00}3. regionserver:{color}
> {color:#00}-Xmx10g -Xms10g -XX:MaxDirectMemorySize=10g
> -XX:MaxGCPauseMillis=150 -XX:MaxTenuringThreshold=2
> -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=5
> -Xloggc:$run_dir/stdout/regionserver_gc_${start_time}.log -Xss256k
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$log_dir -verbose:gc
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintHeapAtGC -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintTenuringDistribution -XX:+PrintSafepointStatistics
> -XX:PrintSafepointStatisticsCount=1 -XX:PrintFLSStatistics=1
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=128m
> -XX:+SafepointTimeout -XX:MonitorBound=16384 -XX:-UseBiasedLocking
> -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=65
> -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=16
> -XX:G1HeapRegionSize=32m -XX:G1MixedGCCountTarget=64
> -XX:G1OldCSetRegionThresholdPercent=5{color}
> {color:#00}block cache is disabled:{color}{color:#00}
> hbase.bucketcache.size
> 0.9
> {color}
>
>Reporter: Gang Xie
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-13639-2.4.diff, HDFS-13639.001.patch,
> HDFS-13639.002.patch, ShortCircuitCache_new_slotReleaser.diff,
> perf_after_improve_SlotReleaser.png, perf_before_improve_SlotReleaser.png
>
>
> When test the performance of the ShortCircuit Read of the HDFS with YCSB, we
> find that SlotReleaser of the ShortCircuitCache has some performance issue.
> The problem is that, the qps of the slot releasing could only reach to 1000+
> while the qps of the slot allocating is ~3000. This means that the replica
> info on datanode could not be released in time, which causes a lot of GCs and
> finally full GCs.
>
> The fireflame graph shows that SlotReleaser spends a lot of time to do domain
> socket connecting and throw/catching the exception when close the domain
> socket and its streams. It doesn't make any sense to do the connecting and
> closing each time. Each time when we connect to the domain socket, Datanode
> allocates a new thread to free the slot. There are a lot of initializing
> work, and it's costly. We need reuse the domain socket.
>
> After switch to reuse the domain socket(see diff attached), we get great
> improvement(see the perf):
> # without reusing the domain socket, the get qps of the YCSB getting worse
> and worse, and after about 45 mins, full GC starts. When we reuse the domain
> socket, no full GC found, and the stress test could be finished smoothly, the
> qps of allocating and releasing match.
> # Due to the datanode young GC, without the improvement, the YCSB get qps is
> even smaller than the one with the improvement, ~3700 VS ~4200.
> The diff is against 2.4, and I think this issue exists till latest version. I
> doesn't have test env with 2.7 and higher version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
[jira] [Updated] (HDFS-14894) Add balancer parameter to balance top used nodes
[ https://issues.apache.org/jira/browse/HDFS-14894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14894: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Add balancer parameter to balance top used nodes > > > Key: HDFS-14894 > URL: https://issues.apache.org/jira/browse/HDFS-14894 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Major > Attachments: HDFS-14894.001.patch > > > We sometimes see a few of our datanodes reach very high usage (due to various > reasons) and we need to reduce their usage in an urgent situation. > We see two ways to achieve it currently, > -Calculate and reset balancing threshold. > -Pick nodes manually according to usage stats and put them in a file and use > `-resource` flag. > However, both of them are not very intuitive or too much manual work in an > urgent close-to-outage situation. Add a small feature to automatically pick > top used hosts will be a straightforward option, for example > `-sourceThreshold 95` to only target datanodes with >95% usage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14718) HttpFS: Sort response by key names as WebHDFS does
[ https://issues.apache.org/jira/browse/HDFS-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14718: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > HttpFS: Sort response by key names as WebHDFS does > -- > > Key: HDFS-14718 > URL: https://issues.apache.org/jira/browse/HDFS-14718 > Project: Hadoop HDFS > Issue Type: Improvement > Components: httpfs >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > > *Example* > See description of HDFS-14665 for an example of LISTSTATUS. > *Analysis* > WebHDFS is [using a > TreeMap|https://github.com/apache/hadoop/blob/99bf1dc9eb18f9b4d0338986d1b8fd2232f1232f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java#L120] > to serialize HdfsFileStatus, while HttpFS [uses a > LinkedHashMap|https://github.com/apache/hadoop/blob/6fcc5639ae32efa5a5d55a6b6cf23af06fc610c3/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java#L107] > to serialize FileStatus. > *Questions* > Why the difference? Is this intentional? > - I looked into the Git history. It seems it's simply because WebHDFS uses > TreeMap from the beginning; and HttpFS uses LinkedHashMap from the beginning. > It is not only limited to LISTSTATUS, but ALL other request's JSON > serialization. > Now the real question: Could/Should we replace ALL LinkedHashMap into TreeMap > in HttpFS serialization in FSOperations class? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7368) Support HDFS specific 'shell' on command 'hdfs dfs' invocation
[ https://issues.apache.org/jira/browse/HDFS-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-7368: --- Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Support HDFS specific 'shell' on command 'hdfs dfs' invocation > -- > > Key: HDFS-7368 > URL: https://issues.apache.org/jira/browse/HDFS-7368 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > Attachments: HDFS-7368-001.patch > > > * *hadoop fs* is the generic implementation for all filesystem > implementations, but some of the operations are supported only in some > filesystems. Ex: snapshot commands, acl commands, xattr commands. > * *hdfs dfs* is recommended in all hdfs related docs in current releases. > In current code both *hdfs shell* and *hadoop fs* points to hadoop common > implementation of FSShell. > It would be better to have HDFS specific extention of FSShell which includes > HDFS only commands in future. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11392) FSPermissionChecker#checkSubAccess should support inodeattribute provider
[
https://issues.apache.org/jira/browse/HDFS-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11392:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> FSPermissionChecker#checkSubAccess should support inodeattribute provider
> -
>
> Key: HDFS-11392
> URL: https://issues.apache.org/jira/browse/HDFS-11392
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: John Zhuge
>Priority: Minor
>
> HDFS-6826 added this TODO in {{FSPermissionChecker#checkSubAccess}}:
> {code:title=FSPermissionChecker#checkSubAccess}
> //TODO have to figure this out with inodeattribute provider
> INodeAttributes inodeAttr =
> getINodeAttrs(components, pathIdx, d, snapshotId);
> {code}
> If inodeattribute provider in play, it always incorrectly returns the root
> attr of the subtree even when it descends multiple levels down the sub tree,
> because the components array is for the root .
> {code:title=FSPermissionChecker#getINodeAttrs}
> private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements,
> inodeAttrs);
> }
> return inodeAttrs;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11381) PreCommit TestDataNodeOutlierDetectionViaMetrics failure
[
https://issues.apache.org/jira/browse/HDFS-11381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11381:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> PreCommit TestDataNodeOutlierDetectionViaMetrics failure
>
>
> Key: HDFS-11381
> URL: https://issues.apache.org/jira/browse/HDFS-11381
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Assignee: Arpit Agarwal
>Priority: Minor
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/18285/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
> {noformat}
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.457 sec <<<
> FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics
> testOutlierIsDetected(org.apache.hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics)
> Time elapsed: 0.27 sec <<< FAILURE!
> java.lang.AssertionError:
> Expected: is <1>
> but: was <0>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:865)
> at org.junit.Assert.assertThat(Assert.java:832)
> at
> org.apache.hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics.testOutlierIsDetected(TestDataNodeOutlierDetectionViaMetrics.java:85)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11039) Expose more configuration properties to hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11039: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Expose more configuration properties to hdfs-default.xml > > > Key: HDFS-11039 > URL: https://issues.apache.org/jira/browse/HDFS-11039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation, newbie >Reporter: Yi Liu >Assignee: Jennica Pounds >Priority: Minor > > There are some configuration properties for hdfs, but have not been exposed > in hdfs-default.xml. > It's convenient for Hadoop user/admin if we add them in the hdfs-default.xml. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11485) HttpFS should warn about weak SSL ciphers
[
https://issues.apache.org/jira/browse/HDFS-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11485:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> HttpFS should warn about weak SSL ciphers
> -
>
> Key: HDFS-11485
> URL: https://issues.apache.org/jira/browse/HDFS-11485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> HDFS-11418 sets a list of default ciphers that contain a few weak ciphers in
> order to maintain backwards compatibility. In addition, users can select weak
> ciphers by env {{HTTPFS_SSL_CIPHERS}}. It'd nice to get warnings about the
> weak ciphers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9821) HDFS configuration should accept friendly time units
[
https://issues.apache.org/jira/browse/HDFS-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-9821:
---
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> HDFS configuration should accept friendly time units
>
>
> Key: HDFS-9821
> URL: https://issues.apache.org/jira/browse/HDFS-9821
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode
>Affects Versions: 2.8.0
>Reporter: Arpit Agarwal
>Assignee: Xiaobing Zhou
>Priority: Major
>
> HDFS configuration keys that define time intervals use units inconsistently
> (Hours, seconds, milliseconds).
> Not all keys have the unit as part of their name. Related keys may use
> different units e.g. {{dfs.blockreport.intervalMsec}} accepts msec while
> {{dfs.blockreport.initialDelay}} accepts seconds. Milliseconds is rarely
> useful as a time unit which makes these values hard to parse when reading
> config files.
> We can either
> # Let existing keys use friendly units e.g. 100ms, 60s, 5m, 1d, 6w etc. This
> can be done compatibly since there will be no conflict with existing valid
> configuration. If no suffix is specified just default to the current time
> unit.
> # Just deprecate the existing keys and define new ones that accept friendly
> units.
> We continue to use fine-grained time units (usually ms) internally in code
> and also accept "ms" option for tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14788) Use dynamic regex filter to ignore copy of source files in Distcp
[ https://issues.apache.org/jira/browse/HDFS-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14788: Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Use dynamic regex filter to ignore copy of source files in Distcp > - > > Key: HDFS-14788 > URL: https://issues.apache.org/jira/browse/HDFS-14788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 3.2.1 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Fix For: 3.3.0 > > > There is a feature in Distcp where we can ignore specific files to get copied > to the destination. This is currently based on a filter regex which is read > from a specific file. The process of creating different regex file for > different distcp jobs seems like a tedious task. What we are proposing is to > expose a regex_filter parameter which can be set during Distcp job creation > and use this filter in a new implementation CopyFilter class. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11203) Rename support during re-encrypt EDEK
[ https://issues.apache.org/jira/browse/HDFS-11203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11203: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Rename support during re-encrypt EDEK > - > > Key: HDFS-11203 > URL: https://issues.apache.org/jira/browse/HDFS-11203 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > > Currently HDFS-10899 disables renames within the EZ if it's under > re-encryption. (similar to current cross-zone rename checks). > We'd like to support rename in the long run, so cluster is fully functioning > during re-encryption. > The reason rename is particularly difficult is: > - We want to re-encrypt all files under an EZ in one pass, without missing any > - We want to iterate through the files and keep track of where we are (i.e. a > cursor), so in case of NN failover/crash, we can resume from fsimage/edits. > - We cannot guarantee namespace is not changed during re-encryption. Newly > created files automatically has new edek, deleted files we don't care. But if > a file is renamed from behind the cursor to before, it may be missed in the > re-encryption. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7702) Move metadata across namenode - Effort to a real distributed namenode
[ https://issues.apache.org/jira/browse/HDFS-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-7702: --- Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Move metadata across namenode - Effort to a real distributed namenode > - > > Key: HDFS-7702 > URL: https://issues.apache.org/jira/browse/HDFS-7702 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ray Zhang >Assignee: Ray Zhang >Priority: Major > Attachments: Namespace Moving Tool Design Proposal.pdf > > > Implement a tool can show in memory namespace tree structure with > weight(size) and a API can move metadata across different namenode. The > purpose is moving data efficiently and faster, without moving blocks on > datanode. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10854) Remove createStripedFile and addBlockToFile by creating real EC files
[
https://issues.apache.org/jira/browse/HDFS-10854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-10854:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Remove createStripedFile and addBlockToFile by creating real EC files
> -
>
> Key: HDFS-10854
> URL: https://issues.apache.org/jira/browse/HDFS-10854
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, test
>Affects Versions: 3.0.0-alpha2
>Reporter: Zhe Zhang
>Assignee: Sammi Chen
>Priority: Major
>
> {{DFSTestUtil#createStripedFile}} and {{addBlockToFile}} were developed
> before we completed EC client. They were used to test the {{NameNode}} EC
> logic when the client was unable to really create/read/write EC files.
> They are causing confusions in other issues about {{NameNode}}. For example,
> in one of the patches under {{HDFS-10301}},
> {{testProcessOverReplicatedAndMissingStripedBlock}} fails because in the test
> we fake a block report from a DN, with a randomly generated storage ID. The
> DN itself is never aware of that storage. This is not possible in a real
> production environment.
> {code}
> DatanodeStorage storage = new
> DatanodeStorage(UUID.randomUUID().toString());
> StorageReceivedDeletedBlocks[] reports = DFSTestUtil
> .makeReportForReceivedBlock(block,
> ReceivedDeletedBlockInfo.BlockStatus.RECEIVING_BLOCK, storage);
> for (StorageReceivedDeletedBlocks report : reports) {
> ns.processIncrementalBlockReport(dn.getDatanodeId(), report);
> }
> {code}
> Now that we have a fully functional EC client, we should remove the old
> testing logic and use similar logic as non-EC tests (creating real files and
> emulate blocks missing / being corrupt).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-3570) Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used space
[
https://issues.apache.org/jira/browse/HDFS-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-3570:
---
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Balancer shouldn't rely on "DFS Space Used %" as that ignores non-DFS used
> space
>
>
> Key: HDFS-3570
> URL: https://issues.apache.org/jira/browse/HDFS-3570
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.0.0-alpha
>Reporter: Harsh J
>Assignee: Akira Ajisaka
>Priority: Minor
> Attachments: HDFS-3570.003.patch, HDFS-3570.2.patch,
> HDFS-3570.aash.1.patch
>
>
> Report from a user here:
> https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pIhNyDVxdVY/b7ENZmEvBjIJ,
> post archived at http://pastebin.com/eVFkk0A0
> This user had a specific DN that had a large non-DFS usage among
> dfs.data.dirs, and very little DFS usage (which is computed against total
> possible capacity).
> Balancer apparently only looks at the usage, and ignores to consider that
> non-DFS usage may also be high on a DN/cluster. Hence, it thinks that if a
> DFS Usage report from DN is 8% only, its got a lot of free space to write
> more blocks, when that isn't true as shown by the case of this user. It went
> on scheduling writes to the DN to balance it out, but the DN simply can't
> accept any more blocks as a result of its disks' state.
> I think it would be better if we _computed_ the actual utilization based on
> {{(100-(actual remaining space))/(capacity)}}, as opposed to the current
> {{(dfs used)/(capacity)}}. Thoughts?
> This isn't very critical, however, cause it is very rare to see DN space
> being used for non DN data, but it does expose a valid bug.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-4319) FSShell copyToLocal creates files with the executable bit set
[
https://issues.apache.org/jira/browse/HDFS-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-4319:
---
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> FSShell copyToLocal creates files with the executable bit set
> -
>
> Key: HDFS-4319
> URL: https://issues.apache.org/jira/browse/HDFS-4319
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.3-alpha
>Reporter: Colin McCabe
>Priority: Minor
>
> With the default value of {{fs.permissions.umask-mode}}, {{022}}, {{FSShell
> copyToLocal}} creates files with the executable bit set.
> If, on the other hand, you change {{fs.permissions.umask-mode}} to something
> like {{133}}, you encounter a different problem. When you use
> {{copyToLocal}} to create directories, they don't have the executable bit
> set, meaning they do not have search permission.
> Since HDFS doesn't allow the executable bit to be set on files, it seems
> illogical to add it in when using {{copyToLocal}}. This is also a
> regression, since branch 1 did not have this problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12308) Erasure Coding: Provide DistributedFileSystem & DFSClient API to return the effective EC policy on a directory or file, including the replication policy
[
https://issues.apache.org/jira/browse/HDFS-12308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12308:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Erasure Coding: Provide DistributedFileSystem & DFSClient API to return the
> effective EC policy on a directory or file, including the replication policy
> -
>
> Key: HDFS-12308
> URL: https://issues.apache.org/jira/browse/HDFS-12308
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Reporter: Sammi Chen
>Assignee: chencan
>Priority: Major
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HADOOP-12308.002.patch, HADOOP-12308.patch,
> HDFS-12308.003.patch, HDFS-12308.004.patch
>
>
> Provide DistributedFileSystem & DFSClient API to return the effective EC
> policy on a directory or file, including the replication policy. Policy name
> will like {{getNominalErasureCodingPolicy(PATH)}} and
> {{getAllNominalErasureCodingPolicies}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-4389) Non-HA DFSClients do not attempt reconnects
[
https://issues.apache.org/jira/browse/HDFS-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-4389:
---
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Non-HA DFSClients do not attempt reconnects
> ---
>
> Key: HDFS-4389
> URL: https://issues.apache.org/jira/browse/HDFS-4389
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, hdfs-client
>Affects Versions: 2.0.0-alpha, 3.0.0-alpha1
>Reporter: Daryn Sharp
>Priority: Major
>
> The HA retry policy implementation appears to have broken non-HA
> {{DFSClient}} connect retries. The ipc
> {{Client.Connection#handleConnectionFailure}} used to perform 45 connection
> attempts, but now it consults a retry policy. For non-HA proxies, the policy
> does not handle {{ConnectException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7902) Expose truncate API for libwebhdfs
[ https://issues.apache.org/jira/browse/HDFS-7902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-7902: --- Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Expose truncate API for libwebhdfs > -- > > Key: HDFS-7902 > URL: https://issues.apache.org/jira/browse/HDFS-7902 > Project: Hadoop HDFS > Issue Type: Improvement > Components: native, webhdfs >Affects Versions: 2.7.0 >Reporter: Yi Liu >Assignee: Yi Liu >Priority: Major > Attachments: HDFS-7902.001.patch, HDFS-7902.002.patch > > > As Colin suggested in HDFS-7838, we will add truncate support for libwebhdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8538) Change the default volume choosing policy to AvailableSpaceVolumeChoosingPolicy
[ https://issues.apache.org/jira/browse/HDFS-8538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-8538: --- Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Change the default volume choosing policy to > AvailableSpaceVolumeChoosingPolicy > --- > > Key: HDFS-8538 > URL: https://issues.apache.org/jira/browse/HDFS-8538 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Major > Attachments: hdfs-8538.001.patch > > > For datanodes with different sized disks, they almost always want the > available space policy. Users with homogenous disks are unaffected. > Since this code has baked for a while, let's change it to be the default. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12859) Admin command resetBalancerBandwidth
[ https://issues.apache.org/jira/browse/HDFS-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12859: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Admin command resetBalancerBandwidth > > > Key: HDFS-12859 > URL: https://issues.apache.org/jira/browse/HDFS-12859 > Project: Hadoop HDFS > Issue Type: New Feature > Components: balancer & mover >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang >Priority: Major > Attachments: > 0003-HDFS-12859-Admin-command-resetBalancerBandwidth.patch, > 0004-HDFS-12859-Admin-command-resetBalancerBandwidth.patch, HDFS-12859.patch > > > We can already set balancer bandwidth dynamically using command > setBalancerBandwidth. The setting value is not persistent and not stored in > configuration file. The different datanodes could their different default or > former setting in configuration. > When we suggested to develop a schedule balancer task which runs at midnight > everyday. We set a larger bandwidth for it and hope to reset the value after > finishing. However, we found it difficult to reset the different setting for > different datanodes as the setBalancerBandwidth command can only set the same > value to all datanodes. If we want to use unique setting for every datanode, > we have to reset the datanodes. > So it would be useful to have a command to synchronize the setting with the > configuration file. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12243) Trash emptier should use Time.monotonicNow()
[ https://issues.apache.org/jira/browse/HDFS-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12243: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Trash emptier should use Time.monotonicNow() > > > Key: HDFS-12243 > URL: https://issues.apache.org/jira/browse/HDFS-12243 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Reporter: Wei-Chiu Chuang >Assignee: Lisheng Sun >Priority: Minor > Attachments: HDFS-12243.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10529) Df reports incorrect usage when appending less than block size
[ https://issues.apache.org/jira/browse/HDFS-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10529: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Df reports incorrect usage when appending less than block size > -- > > Key: HDFS-10529 > URL: https://issues.apache.org/jira/browse/HDFS-10529 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.2, 3.0.0-alpha1 >Reporter: Pranav Prakash >Assignee: Pranav Prakash >Priority: Minor > Labels: datanode, fs, hdfs > Attachments: HDFS-10529.000.patch > > > Steps to recreate issue: > 1. Create a 100MB file on HDFS cluster with 128MB blocksize and replication > factor 3 > 2. Append 100MB to the file > 3. Df reports around 900MB even though it should only be around 600MB. > Looking at the blocks confirms that df is incorrect, as there exist only two > blocks on each DN -- a 128MB block and a 72MB block. > This issue seems to arise because BlockPoolSlice does not account for the > delta increase in dfsUsage when an append happens to a partially-filled > block, and instead naively adds the total block size. For instance, in the > example scenario when when block is "filled" from 100 to 128MB, > addFinalizedBlock() in BlockPoolSlice adds the size of the newly created > block into the total instead of accounting for the difference/delta in block > size between old and new. This has the effect of double-counting the old > partially-filled block: it is counted once when it is first created (in the > example scenario when the 100MB file is created) and again when it becomes > part of the filled block (in the example scenario when the 128MB block is > formed form the initial 100MB block). Thus the perceived size becomes 100MB + > 128MB + 72 = 300 MB for each DN, or 900MB across the cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8115) Make PermissionStatusFormat public
[
https://issues.apache.org/jira/browse/HDFS-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-8115:
---
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Make PermissionStatusFormat public
> --
>
> Key: HDFS-8115
> URL: https://issues.apache.org/jira/browse/HDFS-8115
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Minor
> Attachments: HDFS-8115.1.patch
>
>
> implementations of {{INodeAttributeProvider}} are required to provide an
> implementation of {{getPermissionLong()}} method. Unfortunately, the long
> permission format is an encoding of the user, group and mode with each field
> converted to int using {{SerialNumberManager}} which is package protected.
> Thus it would be nice to make the {{PermissionStatusFormat}} enum public (and
> also make the {{toLong()}} static method public) so that user specified
> implementations of {{INodeAttributeProvider}} may use it.
> This would also make it more consistent with {{AclStatusFormat}} which I
> guess has been made public for the same reason.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11423) Allow ReplicaInfo to be subclassed outside of package
[
https://issues.apache.org/jira/browse/HDFS-11423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11423:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Allow ReplicaInfo to be subclassed outside of package
> -
>
> Key: HDFS-11423
> URL: https://issues.apache.org/jira/browse/HDFS-11423
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.0.0-alpha2
>Reporter: Joe Pallas
>Priority: Minor
> Attachments: HDFS-11423.001.patch
>
>
> The constructor for {{ReplicaInfo}} is package-scope instead of protected.
> This means that an alternative dataset implementation in its own package
> cannot create Replica classes that inherit from ReplicaInfo the way
> {{LocalReplica}} does (or the way {{ProvidedReplica}} does in HDFS-9806).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12122) TestEditLogTailer is flaky
[
https://issues.apache.org/jira/browse/HDFS-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12122:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> TestEditLogTailer is flaky
> --
>
> Key: HDFS-12122
> URL: https://issues.apache.org/jira/browse/HDFS-12122
> Project: Hadoop HDFS
> Issue Type: Test
> Components: namenode
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Priority: Minor
> Labels: flaky-test
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/20225/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
> {noformat}
> Tests run: 12, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 46.901 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
> testTriggersLogRollsForAllStandbyNN[0](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 10.728 sec <<< FAILURE!
> java.lang.AssertionError: Test resulted in an unexpected exit
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testTriggersLogRollsForAllStandbyNN(TestEditLogTailer.java:245)
> testNN1TriggersLogRolls[1](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 0.394 sec <<< FAILURE!
> java.lang.AssertionError: Test resulted in an unexpected exit
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN1TriggersLogRolls(TestEditLogTailer.java:154)
> testNN2TriggersLogRolls[1](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 1.287 sec <<< FAILURE!
> java.lang.AssertionError: Test resulted in an unexpected exit
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN2TriggersLogRolls(TestEditLogTailer.java:159)
> testNN0TriggersLogRolls[1](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 1.286 sec <<< FAILURE!
> java.lang.AssertionError: Test resulted in an unexpected exit
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:201)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN0TriggersLogRolls(TestEditLogTailer.java:149)
> testTriggersLogRollsForAllStandbyNN[1](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 10.469 sec <<< FAILURE!
> java.lang.AssertionError: Test resulted in an unexpected exit
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testTriggersLogRollsForAllStandbyNN(TestEditLogTailer.java:245)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11455) Fix javac warnings in HDFS that caused by deprecated FileSystem APIs
[
https://issues.apache.org/jira/browse/HDFS-11455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11455:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Fix javac warnings in HDFS that caused by deprecated FileSystem APIs
>
>
> Key: HDFS-11455
> URL: https://issues.apache.org/jira/browse/HDFS-11455
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11455.001.patch, HDFS-11455.002.patch,
> HDFS-11455.003.patch
>
>
> There are many javac warnings coming out after FileSystem APIs which promote
> inefficient call patterns being deprecated in HADOOP-13321. The relevant
> warnings:
> {code}
> [WARNING]
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java:[320,18]
> [deprecation] isFile(Path) in FileSystem has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java:[1409,18]
> [deprecation] isFile(Path) in FileSystem has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java:[778,19]
> [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java:[787,20]
> [deprecation] isDirectory(Path) in FileSystem has been deprecated
> [WARNING]
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaByStorageType.java:[834,18]
> [deprecation] isFile(Path) in FileSystem has been
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15140) Replace FoldedTreeSet in Datanode with SortedSet or TreeMap
[ https://issues.apache.org/jira/browse/HDFS-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-15140: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Replace FoldedTreeSet in Datanode with SortedSet or TreeMap > --- > > Key: HDFS-15140 > URL: https://issues.apache.org/jira/browse/HDFS-15140 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.3.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Attachments: HDFS-15140.001.patch, HDFS-15140.002.patch > > > Based on the problems discussed in HDFS-15131, I would like to explore > replacing the FoldedTreeSet structure in the datanode with a builtin Java > equivalent - either SortedSet or TreeMap. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11543) Test multiple erasure coding implementations
[ https://issues.apache.org/jira/browse/HDFS-11543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11543: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Test multiple erasure coding implementations > > > Key: HDFS-11543 > URL: https://issues.apache.org/jira/browse/HDFS-11543 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha2 >Reporter: László Bence Nagy >Priority: Minor > Labels: test > > Potentially, multiple native erasure coding plugins will be available to be > used from HDFS later on. These plugins should be tested as well. For example, > the *NativeRSRawErasureCoderFactory* class - which is used for instantiating > the native ISA-L plugin's encoder and decoder objects - are used in 5 test > files under the > *hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/* > directory. The files are: > - *TestDFSStripedInputStream.java* > - *TestDFSStripedOutputStream.java* > - *TestDFSStripedOutputStreamWithFailure.java* > - *TestReconstructStripedFile.java* > - *TestUnsetAndChangeDirectoryEcPolicy.java* > Other erasure coding plugins should be tested in these cases as well in a > nice way (not by for example making a new file for every new erasure coding > plugin). For this purpose [parameterized > tests|https://github.com/junit-team/junit4/wiki/parameterized-tests] might be > used. > This is also true for the > *hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/erasurecode/rawcoder/* > directory where this approach could be used for example for the > interoperability tests (when it is checked that certain erasure coding > implementations are compatible with each other by doing the encoding and > decoding operations with different plugins and verifying their results). The > plugin pairs which should be tested could be the parameters for the > parameterized tests. > The parameterized test is just an idea, there can be other solutions as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
[ https://issues.apache.org/jira/browse/HDFS-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-7408: --- Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Add a counter in the log that shows the number of block reports processed > - > > Key: HDFS-7408 > URL: https://issues.apache.org/jira/browse/HDFS-7408 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Suresh Srinivas >Assignee: Surendra Singh Lilhore >Priority: Major > Attachments: HDFS-7408.001.patch > > > It would be great to have in the info log corresponding to block report > processing, printing information on how many block reports have been > processed. This can be useful to debug when namenode is unresponsive > especially during startup time to understand if datanodes are sending block > reports multiple times. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11489) KMS should warning about weak SSL ciphers
[
https://issues.apache.org/jira/browse/HDFS-11489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11489:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> KMS should warning about weak SSL ciphers
> -
>
> Key: HDFS-11489
> URL: https://issues.apache.org/jira/browse/HDFS-11489
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.9.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> HADOOP-14083 sets a list of default ciphers that contain a few weak ciphers
> in order to maintain backwards compatibility. In addition, users can select
> weak ciphers by env {{KMS_SSL_CIPHERS}}. It'd nice to get warnings about the
> weak ciphers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11356) figure out what to do about hadoop-hdfs-project/hadoop-hdfs/src/main/native
[ https://issues.apache.org/jira/browse/HDFS-11356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11356: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > figure out what to do about hadoop-hdfs-project/hadoop-hdfs/src/main/native > --- > > Key: HDFS-11356 > URL: https://issues.apache.org/jira/browse/HDFS-11356 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HDFS-11356.001.patch > > > The move of code to hdfs-client-native creation caused all sorts of loose > ends, and this is just another one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14305) Serial number in BlockTokenSecretManager could overlap between different namenodes
[
https://issues.apache.org/jira/browse/HDFS-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-14305:
Target Version/s: 3.4.0 (was: 3.3.0, 3.1.4, 3.2.2, 2.10.1)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Serial number in BlockTokenSecretManager could overlap between different
> namenodes
> --
>
> Key: HDFS-14305
> URL: https://issues.apache.org/jira/browse/HDFS-14305
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, security
>Reporter: Chao Sun
>Assignee: Konstantin Shvachko
>Priority: Major
> Labels: multi-sbnn
> Attachments: HDFS-14305-007.patch, HDFS-14305-008.patch,
> HDFS-14305.001.patch, HDFS-14305.002.patch, HDFS-14305.003.patch,
> HDFS-14305.004.patch, HDFS-14305.005.patch, HDFS-14305.006.patch
>
>
> Currently, a {{BlockTokenSecretManager}} starts with a random integer as the
> initial serial number, and then use this formula to rotate it:
> {code:java}
> this.intRange = Integer.MAX_VALUE / numNNs;
> this.nnRangeStart = intRange * nnIndex;
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> while {{numNNs}} is the total number of NameNodes in the cluster, and
> {{nnIndex}} is the index of the current NameNode specified in the
> configuration {{dfs.ha.namenodes.}}.
> However, with this approach, different NameNode could have overlapping ranges
> for serial number. For simplicity, let's assume {{Integer.MAX_VALUE}} is 100,
> and we have 2 NameNodes {{nn1}} and {{nn2}} in configuration. Then the ranges
> for these two are:
> {code}
> nn1 -> [-49, 49]
> nn2 -> [1, 99]
> {code}
> This is because the initial serial number could be any negative integer.
> Moreover, when the keys are updated, the serial number will again be updated
> with the formula:
> {code}
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> which means the new serial number could be updated to a range that belongs to
> a different NameNode, thus increasing the chance of collision again.
> When the collision happens, DataNodes could overwrite an existing key which
> will cause clients to fail because of {{InvalidToken}} error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8065) Erasure coding: Support truncate at striped group boundary
[ https://issues.apache.org/jira/browse/HDFS-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-8065: --- Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Erasure coding: Support truncate at striped group boundary > -- > > Key: HDFS-8065 > URL: https://issues.apache.org/jira/browse/HDFS-8065 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Yi Liu >Assignee: Rakesh Radhakrishnan >Priority: Major > Attachments: HDFS-8065-00.patch, HDFS-8065-01.patch > > > We can support truncate at striped group boundary firstly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11672) Fix some misleading log messages related to EC block groups
[
https://issues.apache.org/jira/browse/HDFS-11672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11672:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Fix some misleading log messages related to EC block groups
> ---
>
> Key: HDFS-11672
> URL: https://issues.apache.org/jira/browse/HDFS-11672
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Manoj Govindassamy
>Priority: Minor
> Labels: newbie
>
> I turned the NameNode's BlockStateChange and hdfs.StateChange logs up to
> debug, noticed some out of date log messages like the following:
> {noformat}
> 2017-04-18 17:19:50,639 [IPC Server handler 4 on 45133] DEBUG
> hdfs.StateChange (FSDirWriteFileOp.java:addBlock(507)) - DIR*
> FSDirectory.addBlock: /test2RecoveryTasksForSameBlockGroup with
> blk_-9223372036854775792_1001 block is added to the in-memory file system
> 2017-04-18 17:19:50,836 [Block report processor] DEBUG BlockStateChange
> (LowRedundancyBlocks.java:update(353)) - BLOCK*
> NameSystem.LowRedundancyBlock.update: blk_-9223372036854775792_1001 has only
> 8 replicas and needs 9 replicas so is added to neededReconstructions at
> priority level 2
> 2017-04-18 17:19:51,235 [IPC Server handler 4 on 45133] DEBUG
> hdfs.StateChange (FSNamesystem.java:closeFile(3740)) - closeFile:
> /test2RecoveryTasksForSameBlockGroup with 1 blocks is persisted to the file
> system
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10616) Improve performance of path handling
[ https://issues.apache.org/jira/browse/HDFS-10616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10616: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Improve performance of path handling > > > Key: HDFS-10616 > URL: https://issues.apache.org/jira/browse/HDFS-10616 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Major > Attachments: 2.6-2.7.1-heap.png > > > Path handling in the namesystem and directory is very inefficient. The path > is repeatedly resolved, decomposed into path components, recombined to a full > path. parsed again, throughout the system. This is directly inefficient for > general performance, and indirectly via unnecessary pressure on young gen GC. > The namesystem should only operate on paths, parse it once into inodes, and > the directory should only operate on inodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14312) KMS-o-meter: Scale test KMS using kms audit log
[ https://issues.apache.org/jira/browse/HDFS-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14312: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > KMS-o-meter: Scale test KMS using kms audit log > --- > > Key: HDFS-14312 > URL: https://issues.apache.org/jira/browse/HDFS-14312 > Project: Hadoop HDFS > Issue Type: New Feature > Components: kms >Affects Versions: 3.3.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > It appears to me that Dynamometer's architecture allows KMS scale tests too. > I imagine there are two ways to scale test a KMS. > # Take KMS audit logs, and replay the logs against a KMS. > # Configure Dynamometer to start KMS in addition to NameNode. Assuming the > fsimage comes from an encrypted cluster, replaying HDFS audit log also tests > KMS. > It would be even more interesting to have a tool that converts uncrypted > cluster fsimage to an encrypted one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11350) Document the missing settings relevant to DataNode disk IO statistics
[
https://issues.apache.org/jira/browse/HDFS-11350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11350:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Document the missing settings relevant to DataNode disk IO statistics
> -
>
> Key: HDFS-11350
> URL: https://issues.apache.org/jira/browse/HDFS-11350
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-11350.001.patch
>
>
> HDFS-11299 and HDFS-11339 introduced some new setting relevant to profiling
> hooks to expose latency statistics around DataNode disk IO. These settings
> should be intended for users but they are not documented. Totally three
> relevant settings:
> 1.dfs.datanode.enable.fileio.profiling
> 2.dfs.datanode.enable.fileio.fault.injection
> 3.dfs.datanode.fileio.profiling.sampling.fraction
> Actually, only the setting {{dfs.datanode.enable.fileio.fault.injection}}
> don't need to be documented.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13656) Logging more info when client completes file error
[
https://issues.apache.org/jira/browse/HDFS-13656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-13656:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Logging more info when client completes file error
> --
>
> Key: HDFS-13656
> URL: https://issues.apache.org/jira/browse/HDFS-13656
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.2
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-13656.001.patch, error-hdfs.png
>
>
> We found the error when dfs client completes file.
> !error-hdfs.png!
> Now the error log is too simple and cannot provide enough info for debugging.
> And this error will failed the write operation and finally lead our critical
> tasks failed. It will be good to print retry time and file path info in
> {{DFSOutputStream#completeFile}}. In addition, the log level can be updated
> from INFO to WARN, it won't be ignored easily by users.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7550) Minor followon cleanups from HDFS-7543
[ https://issues.apache.org/jira/browse/HDFS-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-7550: --- Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Minor followon cleanups from HDFS-7543 > -- > > Key: HDFS-7550 > URL: https://issues.apache.org/jira/browse/HDFS-7550 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.7.0 >Reporter: Charles Lamb >Priority: Minor > Attachments: HDFS-7550.001.patch > > > The commit of HDFS-7543 crossed paths with these comments: > FSDirMkdirOp.java > in #mkdirs, you removed the final String srcArg = src. This should be left > in. Many IDEs will whine about making assignments to formal args and that's > why it was put in in the first place. > FSDirRenameOp.java > #renameToInt, dstIIP (and resultingStat) could benefit from final's. > FSDirXAttrOp.java > I'm not sure why you've moved the call to getINodesInPath4Write and > checkXAttrChangeAccess inside the writeLock. > FSDirStatAndListing.java > The javadoc for the @param src needs to be changed to reflect that it's an > INodesInPath, not a String. Nit: it might be better to rename the > INodesInPath arg from src to iip. > #getFileInfo4DotSnapshot is now unused since you in-lined it into > #getFileInfo. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14528) Failover from Active to Standby Failed
[ https://issues.apache.org/jira/browse/HDFS-14528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-14528: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Failover from Active to Standby Failed > > > Key: HDFS-14528 > URL: https://issues.apache.org/jira/browse/HDFS-14528 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Reporter: Ravuri Sushma sree >Assignee: Ravuri Sushma sree >Priority: Major > Labels: multi-sbnn > Attachments: HDFS-14528.003.patch, HDFS-14528.004.patch, > HDFS-14528.005.patch, HDFS-14528.006.patch, HDFS-14528.007.patch, > HDFS-14528.2.Patch, ZKFC_issue.patch > > > *In a cluster with more than one Standby namenode, manual failover throws > exception for some cases* > *When trying to exectue the failover command from active to standby* > *._/hdfs haadmin -failover nn1 nn2, below Exception is thrown_* > Operation failed: Call From X-X-X-X/X-X-X-X to Y-Y-Y-Y: failed on > connection exception: java.net.ConnectException: Connection refused > This is encountered in the following cases : > Scenario 1 : > Namenodes - NN1(Active) , NN2(Standby), NN3(Standby) > When trying to manually failover from NN1 to NN2 if NN3 is down, Exception is > thrown > Scenario 2 : > Namenodes - NN1(Active) , NN2(Standby), NN3(Standby) > ZKFC's - ZKFC1, ZKFC2, ZKFC3 > When trying to manually failover using NN1 to NN3 if NN3's ZKFC (ZKFC3) is > down, Exception is thrown -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12257) Expose getSnapshottableDirListing as a public API in HdfsAdmin
[ https://issues.apache.org/jira/browse/HDFS-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-12257: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Expose getSnapshottableDirListing as a public API in HdfsAdmin > -- > > Key: HDFS-12257 > URL: https://issues.apache.org/jira/browse/HDFS-12257 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 2.6.5 >Reporter: Andrew Wang >Assignee: Huafeng Wang >Priority: Major > Attachments: HDFS-12257.001.patch, HDFS-12257.002.patch, > HDFS-12257.003.patch > > > Found at HIVE-16294. We have a CLI API for listing snapshottable dirs, but no > programmatic API. Other snapshot APIs are exposed in HdfsAdmin, I think we > should expose listing there as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10743) MiniDFSCluster test runtimes can be drastically reduce
[
https://issues.apache.org/jira/browse/HDFS-10743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-10743:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> MiniDFSCluster test runtimes can be drastically reduce
> --
>
> Key: HDFS-10743
> URL: https://issues.apache.org/jira/browse/HDFS-10743
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.0.0-alpha
>Reporter: Daryn Sharp
>Assignee: Kuhu Shukla
>Priority: Major
> Attachments: HDFS-10743.001.patch, HDFS-10743.002.patch,
> HDFS-10743.003.patch
>
>
> {{MiniDFSCluster}} tests have excessive runtimes. The main problem appears
> to be the heartbeat interval. The NN may have to wait up to 3s (default
> value) for all DNs to heartbeat, triggering registration, so NN can go
> active. Tests that repeatedly restart the NN are severely affected.
> Example for varying heartbeat intervals for {{TestFSImageWithAcl}}:
> * 3s = ~70s -- (disgusting, why I investigated)
> * 1s = ~27s
> * 500ms = ~17s -- (had to hack DNConf for millisecond precision)
> That a 4x improvement in runtime.
> 17s is still excessively long for what the test does. Further areas to
> explore when running tests:
> * Reduce numerous sleeps intervals in DN's {{BPServiceActor}}.
> * Ensure heartbeats and initial BR are sent immediately upon (re)registration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10429) DataStreamer interrupted warning always appears when using CLI upload file
[
https://issues.apache.org/jira/browse/HDFS-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-10429:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> DataStreamer interrupted warning always appears when using CLI upload file
> ---
>
> Key: HDFS-10429
> URL: https://issues.apache.org/jira/browse/HDFS-10429
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Zhiyuan Yang
>Priority: Minor
> Attachments: HDFS-10429.1.patch, HDFS-10429.2.patch,
> HDFS-10429.3.patch
>
>
> Every time I use 'hdfs dfs -put' upload file, this warning is printed:
> {code:java}
> 16/05/18 20:57:56 WARN hdfs.DataStreamer: Caught exception
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Thread.join(Thread.java:1245)
> at java.lang.Thread.join(Thread.java:1319)
> at
> org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:871)
> at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:519)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:696)
> {code}
> The reason is this: originally, DataStreamer::closeResponder always prints a
> warning about InterruptedException; since HDFS-9812,
> DFSOutputStream::closeImpl always forces threads to close, which causes
> InterruptedException.
> A simple fix is to use debug level log instead of warning level.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10911) Change edit log OP_UPDATE_BLOCKS to store delta blocks only.
[
https://issues.apache.org/jira/browse/HDFS-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-10911:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Change edit log OP_UPDATE_BLOCKS to store delta blocks only.
>
>
> Key: HDFS-10911
> URL: https://issues.apache.org/jira/browse/HDFS-10911
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.7.3, 3.0.0-alpha1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
>
> Every time a HDFS client {{close}} or {{hflush}} an open file, NameNode
> enumerates all the blocks and stores then into edit log (OP_UPDATE_BLOCKS).
> It would cause problem when the client is appending a large file frequently
> (i.e., WAL).
> Because HDFS is append only, we can only store the blocks that have been
> changed (delta blocks) in edit log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13495) RBF: Support Router Admin REST API
[ https://issues.apache.org/jira/browse/HDFS-13495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-13495: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > RBF: Support Router Admin REST API > -- > > Key: HDFS-13495 > URL: https://issues.apache.org/jira/browse/HDFS-13495 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Mohammad Arshad >Assignee: Fengnan Li >Priority: Major > Labels: RBF > Attachments: HDFS-13495-001.patch > > > This JIRA intends to add REST API support for all admin commands. Router > Admin REST APIs can be useful in managing the Routers from a central > management layer tool. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11232) System.err should be System.out
[
https://issues.apache.org/jira/browse/HDFS-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11232:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> System.err should be System.out
> ---
>
> Key: HDFS-11232
> URL: https://issues.apache.org/jira/browse/HDFS-11232
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ethan Li
>Priority: Trivial
> Attachments: HDFS-11232.001.patch, HDFS-11232.002.patch
>
>
> In
> /Users/Ethan/Worksplace/IntelliJWorkspace/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java,
> System.err.println("Generating new cluster id:"); is used. I think it should
> be System.out.println(...) since this is not an error message
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11885) createEncryptionZone should not block on initializing EDEK cache
[
https://issues.apache.org/jira/browse/HDFS-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11885:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> createEncryptionZone should not block on initializing EDEK cache
>
>
> Key: HDFS-11885
> URL: https://issues.apache.org/jira/browse/HDFS-11885
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Affects Versions: 2.6.5
>Reporter: Andrew Wang
>Assignee: Andrew Wang
>Priority: Major
> Attachments: HDFS-11885.001.patch, HDFS-11885.002.patch,
> HDFS-11885.003.patch, HDFS-11885.004.patch
>
>
> When creating an encryption zone, we call {{ensureKeyIsInitialized}}, which
> calls {{provider.warmUpEncryptedKeys(keyName)}}. This is a blocking call,
> which attempts to fill the key cache up to the low watermark.
> If the KMS is down or slow, this can take a very long time, and cause the
> createZone RPC to fail with a timeout.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11464) Improve the selection in choosing storage for blocks
[
https://issues.apache.org/jira/browse/HDFS-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11464:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Improve the selection in choosing storage for blocks
>
>
> Key: HDFS-11464
> URL: https://issues.apache.org/jira/browse/HDFS-11464
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-11464.001.patch, HDFS-11464.002.patch,
> HDFS-11464.003.patch, HDFS-11464.004.patch, HDFS-11464.005.patch
>
>
> Currently the logic in choosing storage for blocks is not a good way. It
> always uses the first valid storage of a given StorageType ({{see
> DataNodeDescriptor#chooseStorage4Block}}). This should not be a good
> selection. That means blcoks will always be written to the same volume (first
> volume) and other valid volumes have no choices. This problem is brought up
> by this comment (
> https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15878382&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15878382
> )
> There is one solution from me:
> * First, based on existing storages in one node, extract all the valid
> storages into a collection.
> * Then, disrupt the order of these vaild storages, get a new collection.
> * Finally, get the first storage from the new storages collection.
> These steps will be executed in {{DataNodeDescriptor#chooseStorage4Block}}
> and replace current logic. I think this improvement can be done as a subtask
> under HDFS-11419. Any further comments are welcomed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-14476:
Target Version/s: 3.4.0
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> lock too long when fix inconsistent blocks between disk and in-memory
> -
>
> Key: HDFS-14476
> URL: https://issues.apache.org/jira/browse/HDFS-14476
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0, 2.7.0, 3.0.3
>Reporter: Sean Chow
>Assignee: Sean Chow
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14476-branch-2.01.patch,
> HDFS-14476-branch-2.02.patch, HDFS-14476-branch-2.10.02.patch,
> HDFS-14476.00.patch, HDFS-14476.002.patch, HDFS-14476.01.patch,
> HDFS-14476.branch-3.2.001.patch, datanode-with-patch-14476.png
>
>
> When directoryScanner have the results of differences between disk and
> in-memory blocks. it will try to run {{checkAndUpdate}} to fix it. However
> {{FsDatasetImpl.checkAndUpdate}} is a synchronized call
> As I have about 6millions blocks for every datanodes and every 6hours' scan
> will have about 25000 abnormal blocks to fix. That leads to a long lock
> holding FsDatasetImpl object.
> let's assume every block need 10ms to fix(because of latency of SAS disk),
> that will cost 250 seconds to finish. That means all reads and writes will be
> blocked for 3mins for that datanode.
>
> {code:java}
> 2019-05-06 08:06:51,704 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing
> metadata files:23574, missing block files:23574, missing blocks in
> memory:47625, mismatched blocks:0
> ...
> 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 588402ms to process 1 commands from NN
> {code}
> Take long time to process command from nn because threads are blocked. And
> namenode will see long lastContact time for this datanode.
> Maybe this affect all hdfs versions.
> *how to fix:*
> just like process invalidate command from namenode with 1000 batch size, fix
> these abnormal block should be handled with batch too and sleep 2 seconds
> between the batch to allow normal reading/writing blocks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12667) KMSClientProvider#ValueQueue does synchronous fetch of edeks in background async thread.
[
https://issues.apache.org/jira/browse/HDFS-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12667:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> KMSClientProvider#ValueQueue does synchronous fetch of edeks in background
> async thread.
>
>
> Key: HDFS-12667
> URL: https://issues.apache.org/jira/browse/HDFS-12667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, kms
>Affects Versions: 3.0.0-alpha4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Attachments: HDFS-12667-001.patch, HDFS-12667-002.patch
>
>
> There are couple of issues in KMSClientProvider#ValueQueue.
> 1.
> {code:title=ValueQueue.java|borderStyle=solid}
> private final LoadingCache> keyQueues;
> // Stripped rwlocks based on key name to synchronize the queue from
> // the sync'ed rw-thread and the background async refill thread.
> private final List lockArray =
> new ArrayList<>(LOCK_ARRAY_SIZE);
> {code}
> It hashes the key name into 16 buckets.
> In the code chunk below,
> {code:title=ValueQueue.java|borderStyle=solid}
> public List getAtMost(String keyName, int num) throws IOException,
> ExecutionException {
> ...
> ...
> readLock(keyName);
> E val = keyQueue.poll();
> readUnlock(keyName);
> ...
> }
> private void submitRefillTask(final String keyName,
> final Queue keyQueue) throws InterruptedException {
> ...
> ...
> writeLock(keyName); // It holds the write lock while the key is
> being asynchronously fetched. So the read requests for all the keys that
> hashes to this bucket will essentially be blocked.
> try {
> if (keyQueue.size() < threshold && !isCanceled()) {
> refiller.fillQueueForKey(name, keyQueue,
> cacheSize - keyQueue.size());
> }
> ...
> } finally {
> writeUnlock(keyName);
> }
> }
> }
> {code}
> According to above code chunk, if two keys (lets say key1 and key2) hashes to
> the same bucket (between 1 and 16), then if key1 is asynchronously being
> refetched then all the getKey for key2 will be blocked.
> 2. Due to stripped rw locks, the asynchronous behavior of refill keys is now
> synchronous to other handler threads.
> I understand that locks were added so that we don't kick off multiple
> asynchronous refilling thread for the same key.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10237) Support specifying checksum type in WebHDFS/HTTPFS writers
[ https://issues.apache.org/jira/browse/HDFS-10237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10237: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Support specifying checksum type in WebHDFS/HTTPFS writers > -- > > Key: HDFS-10237 > URL: https://issues.apache.org/jira/browse/HDFS-10237 > Project: Hadoop HDFS > Issue Type: New Feature > Components: webhdfs >Affects Versions: 2.8.0 >Reporter: Harsh J >Priority: Minor > Attachments: HDFS-10237.000.patch, HDFS-10237.001.patch, > HDFS-10237.002.patch, HDFS-10237.002.patch > > > Currently you cannot set a desired checksum type over a WebHDFS or HTTPFS > writer, as you can with the regular DFS writer (done via HADOOP-8240) > This JIRA covers the changes necessary to bring the same ability to WebHDFS > and HTTPFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9940) Balancer should not use property dfs.datanode.balance.max.concurrent.moves
[
https://issues.apache.org/jira/browse/HDFS-9940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-9940:
---
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> Balancer should not use property dfs.datanode.balance.max.concurrent.moves
> --
>
> Key: HDFS-9940
> URL: https://issues.apache.org/jira/browse/HDFS-9940
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Labels: supportability
>
> It is very confusing for both Balancer and Datanode to use the same property
> {{dfs.datanode.balance.max.concurrent.moves}}. It is especially so for the
> Balancer because the property has "datanode" in the name string. Many
> customers forget to set the property for the Balancer.
> Change the Balancer to use a new property
> {{dfs.balancer.max.concurrent.moves}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12811) A cheaper, faster, less memory intensive Hadoop fsck
[
https://issues.apache.org/jira/browse/HDFS-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12811:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> A cheaper, faster, less memory intensive Hadoop fsck
>
>
> Key: HDFS-12811
> URL: https://issues.apache.org/jira/browse/HDFS-12811
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
>
> A cheaper, faster, less memory intensive, approach is eliminate the traversal
> by directly scanning the inode table.
> A side-effect is paths would be scanned and displayed in a random order. A
> new option, ex. "-direct" or "-fast", would likely be required to avoid
> compatibility issues.
> PS: This enhancement is valid only if the path is root directory {{/}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10848) Move hadoop-hdfs-native-client module into hadoop-hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10848: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Move hadoop-hdfs-native-client module into hadoop-hdfs-client > - > > Key: HDFS-10848 > URL: https://issues.apache.org/jira/browse/HDFS-10848 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Akira Ajisaka >Assignee: Huafeng Wang >Priority: Major > Attachments: HDFS-10848.001.patch > > > When a patch changes hadoop-hdfs-client module, Jenkins does not pick up the > tests in the native code. That way we overlooked test failure when committing > the patch. (ex. HDFS-10844) > [~aw] said in HDFS-10844, > bq. Ideally, all of this native code would be hdfs-client. Then when a change > is made to to that code, this code will also get tested. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10364) Log current node in reversexml tool when parse failed
[ https://issues.apache.org/jira/browse/HDFS-10364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-10364: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Log current node in reversexml tool when parse failed > - > > Key: HDFS-10364 > URL: https://issues.apache.org/jira/browse/HDFS-10364 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Trivial > Attachments: HDFS-10364.01.patch > > > Sometimes we want to modify the xml before converting it. If some error > happened, it's hard to find out. Adding a line to tell where the failure is > would be helpful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12116) BlockReportTestBase#blockReport_08 and #blockReport_08 intermittently fail
[
https://issues.apache.org/jira/browse/HDFS-12116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12116:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> BlockReportTestBase#blockReport_08 and #blockReport_08 intermittently fail
> --
>
> Key: HDFS-12116
> URL: https://issues.apache.org/jira/browse/HDFS-12116
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 0.22.0
>Reporter: Xiao Chen
>Assignee: Zsolt Venczel
>Priority: Major
> Attachments: HDFS-12116.01.patch, HDFS-12116.02.patch,
> HDFS-12116.03.patch,
> TEST-org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage.xml
>
>
> This seems to be long-standing, but the failure rate (~10%) is slightly
> higher in dist-test run in using cdh.
> In both _08 and _09 tests:
> # an attempt is made to make a replica in {{TEMPORARY}}
> state, by {{waitForTempReplica}}.
> # Once that's returned, the test goes on to verify block reports shows
> correct pending replication blocks.
> But there's a race condition. If the replica is replicated between steps #1
> and #2, {{getPendingReplicationBlocks}} could return 0 or 1, depending on how
> many replicas are replicated, hence failing the test.
> Failures are seen on both {{TestNNHandlesBlockReportPerStorage}} and
> {{TestNNHandlesCombinedBlockReport}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11987) DistributedFileSystem#create and append do not honor CreateFlag.CREATE|APPEND
[
https://issues.apache.org/jira/browse/HDFS-11987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-11987:
Target Version/s: 3.4.0 (was: 3.3.0)
Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a
blocker.
> DistributedFileSystem#create and append do not honor CreateFlag.CREATE|APPEND
> -
>
> Key: HDFS-11987
> URL: https://issues.apache.org/jira/browse/HDFS-11987
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.8.1, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HDFS-11987.00.patch
>
>
> {{DistributedFileSystem#create()}} and {{DistributedFIleSystem#append()}} do
> not honor the expected behavior on {{CreateFlag.CREATE|APPEND}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11825) Make dfshealth datanode information more readable
[ https://issues.apache.org/jira/browse/HDFS-11825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated HDFS-11825: Target Version/s: 3.4.0 (was: 3.3.0) Bulk update: moved all 3.3.0 non-blocker issues, please move back if it is a blocker. > Make dfshealth datanode information more readable > - > > Key: HDFS-11825 > URL: https://issues.apache.org/jira/browse/HDFS-11825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0-alpha2 >Reporter: Sarah Victor >Assignee: Sarah Victor >Priority: Trivial > Attachments: HADOOP-11825.1.patch > > > On the dfshealth.html#tab-datanode web page to monitor health of Hadoop nodes > (hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html) > (1) The Capacity column reports the node capacity but is sorted by another > value that is not reported (ie) the % Capacity Used. It will be good to name > the column heading as "Capacity Used [%]" and report both the Capacity Used > and the Capacity Used % > (2) The Block Pool Used column reports the block pools used but is sorted by > the % Block Pool Used, It will be good to name the column heading as Block > Pool Used [%] > (3) The "Last contact" column is more easily understood if we call it "Last > Heartbeat". Adding the measurement unit [secs] to the header makes it more > easily understood. > (3) Adding the measurement unit [mins] to the "Last Block Report" header > makes it more easily understood. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12667) KMSClientProvider#ValueQueue does synchronous fetch of edeks in background async thread.
[
https://issues.apache.org/jira/browse/HDFS-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081492#comment-17081492
]
Hadoop QA commented on HDFS-12667:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-12667 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12667 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12895068/HDFS-12667-002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29113/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> KMSClientProvider#ValueQueue does synchronous fetch of edeks in background
> async thread.
>
>
> Key: HDFS-12667
> URL: https://issues.apache.org/jira/browse/HDFS-12667
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, kms
>Affects Versions: 3.0.0-alpha4
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Attachments: HDFS-12667-001.patch, HDFS-12667-002.patch
>
>
> There are couple of issues in KMSClientProvider#ValueQueue.
> 1.
> {code:title=ValueQueue.java|borderStyle=solid}
> private final LoadingCache> keyQueues;
> // Stripped rwlocks based on key name to synchronize the queue from
> // the sync'ed rw-thread and the background async refill thread.
> private final List lockArray =
> new ArrayList<>(LOCK_ARRAY_SIZE);
> {code}
> It hashes the key name into 16 buckets.
> In the code chunk below,
> {code:title=ValueQueue.java|borderStyle=solid}
> public List getAtMost(String keyName, int num) throws IOException,
> ExecutionException {
> ...
> ...
> readLock(keyName);
> E val = keyQueue.poll();
> readUnlock(keyName);
> ...
> }
> private void submitRefillTask(final String keyName,
> final Queue keyQueue) throws InterruptedException {
> ...
> ...
> writeLock(keyName); // It holds the write lock while the key is
> being asynchronously fetched. So the read requests for all the keys that
> hashes to this bucket will essentially be blocked.
> try {
> if (keyQueue.size() < threshold && !isCanceled()) {
> refiller.fillQueueForKey(name, keyQueue,
> cacheSize - keyQueue.size());
> }
> ...
> } finally {
> writeUnlock(keyName);
> }
> }
> }
> {code}
> According to above code chunk, if two keys (lets say key1 and key2) hashes to
> the same bucket (between 1 and 16), then if key1 is asynchronously being
> refetched then all the getKey for key2 will be blocked.
> 2. Due to stripped rw locks, the asynchronous behavior of refill keys is now
> synchronous to other handler threads.
> I understand that locks were added so that we don't kick off multiple
> asynchronous refilling thread for the same key.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12859) Admin command resetBalancerBandwidth
[
https://issues.apache.org/jira/browse/HDFS-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081495#comment-17081495
]
Hadoop QA commented on HDFS-12859:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-12859 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12859 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12899783/0004-HDFS-12859-Admin-command-resetBalancerBandwidth.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29110/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Admin command resetBalancerBandwidth
>
>
> Key: HDFS-12859
> URL: https://issues.apache.org/jira/browse/HDFS-12859
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: balancer & mover
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments:
> 0003-HDFS-12859-Admin-command-resetBalancerBandwidth.patch,
> 0004-HDFS-12859-Admin-command-resetBalancerBandwidth.patch, HDFS-12859.patch
>
>
> We can already set balancer bandwidth dynamically using command
> setBalancerBandwidth. The setting value is not persistent and not stored in
> configuration file. The different datanodes could their different default or
> former setting in configuration.
> When we suggested to develop a schedule balancer task which runs at midnight
> everyday. We set a larger bandwidth for it and hope to reset the value after
> finishing. However, we found it difficult to reset the different setting for
> different datanodes as the setBalancerBandwidth command can only set the same
> value to all datanodes. If we want to use unique setting for every datanode,
> we have to reset the datanodes.
> So it would be useful to have a command to synchronize the setting with the
> configuration file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7368) Support HDFS specific 'shell' on command 'hdfs dfs' invocation
[
https://issues.apache.org/jira/browse/HDFS-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081491#comment-17081491
]
Hadoop QA commented on HDFS-7368:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-7368 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-7368 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12679814/HDFS-7368-001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29116/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Support HDFS specific 'shell' on command 'hdfs dfs' invocation
> --
>
> Key: HDFS-7368
> URL: https://issues.apache.org/jira/browse/HDFS-7368
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Attachments: HDFS-7368-001.patch
>
>
> * *hadoop fs* is the generic implementation for all filesystem
> implementations, but some of the operations are supported only in some
> filesystems. Ex: snapshot commands, acl commands, xattr commands.
> * *hdfs dfs* is recommended in all hdfs related docs in current releases.
> In current code both *hdfs shell* and *hadoop fs* points to hadoop common
> implementation of FSShell.
> It would be better to have HDFS specific extention of FSShell which includes
> HDFS only commands in future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11464) Improve the selection in choosing storage for blocks
[
https://issues.apache.org/jira/browse/HDFS-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081493#comment-17081493
]
Hadoop QA commented on HDFS-11464:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-11464 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-11464 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12867091/HDFS-11464.005.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29111/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Improve the selection in choosing storage for blocks
>
>
> Key: HDFS-11464
> URL: https://issues.apache.org/jira/browse/HDFS-11464
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-11464.001.patch, HDFS-11464.002.patch,
> HDFS-11464.003.patch, HDFS-11464.004.patch, HDFS-11464.005.patch
>
>
> Currently the logic in choosing storage for blocks is not a good way. It
> always uses the first valid storage of a given StorageType ({{see
> DataNodeDescriptor#chooseStorage4Block}}). This should not be a good
> selection. That means blcoks will always be written to the same volume (first
> volume) and other valid volumes have no choices. This problem is brought up
> by this comment (
> https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15878382&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15878382
> )
> There is one solution from me:
> * First, based on existing storages in one node, extract all the valid
> storages into a collection.
> * Then, disrupt the order of these vaild storages, get a new collection.
> * Finally, get the first storage from the new storages collection.
> These steps will be executed in {{DataNodeDescriptor#chooseStorage4Block}}
> and replace current logic. I think this improvement can be done as a subtask
> under HDFS-11419. Any further comments are welcomed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10848) Move hadoop-hdfs-native-client module into hadoop-hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081496#comment-17081496
]
Hadoop QA commented on HDFS-10848:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} HDFS-10848 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10848 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861393/HDFS-10848.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29122/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Move hadoop-hdfs-native-client module into hadoop-hdfs-client
> -
>
> Key: HDFS-10848
> URL: https://issues.apache.org/jira/browse/HDFS-10848
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Akira Ajisaka
>Assignee: Huafeng Wang
>Priority: Major
> Attachments: HDFS-10848.001.patch
>
>
> When a patch changes hadoop-hdfs-client module, Jenkins does not pick up the
> tests in the native code. That way we overlooked test failure when committing
> the patch. (ex. HDFS-10844)
> [~aw] said in HDFS-10844,
> bq. Ideally, all of this native code would be hdfs-client. Then when a change
> is made to to that code, this code will also get tested.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
