[jira] [Created] (HDFS-15388) DFS cacheadmin, ECAdmin, StoragePolicyAdmin commands should handle ViewFSOverloadScheme
Uma Maheswara Rao G created HDFS-15388: -- Summary: DFS cacheadmin, ECAdmin, StoragePolicyAdmin commands should handle ViewFSOverloadScheme Key: HDFS-15388 URL: https://issues.apache.org/jira/browse/HDFS-15388 Project: Hadoop HDFS Issue Type: Sub-task Components: viewfs Affects Versions: 3.2.1 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G There are some more DFS specific admin tools, which should handle ViewFSOverloadScheme when scheme is hdfs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15387) FSUsage$DF should consider ViewFSOverloadScheme in processPath
Uma Maheswara Rao G created HDFS-15387: -- Summary: FSUsage$DF should consider ViewFSOverloadScheme in processPath Key: HDFS-15387 URL: https://issues.apache.org/jira/browse/HDFS-15387 Project: Hadoop HDFS Issue Type: Sub-task Components: viewfs Affects Versions: 3.2.1 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Currently for calculating DF, processPath checks if it's ViewFS scheme, it gets status from all fs and calculate. If not it will directly call fs.getStatus. Here we should treat ViewFSOverloadScheme also in ViewFS flow -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15346) RBF: DistCpFedBalance implementation
[ https://issues.apache.org/jira/browse/HDFS-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126332#comment-17126332 ] Yiqun Lin commented on HDFS-15346: -- Will give detailed review on this weekend, [~LiJinglun]. > RBF: DistCpFedBalance implementation > > > Key: HDFS-15346 > URL: https://issues.apache.org/jira/browse/HDFS-15346 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15346.001.patch, HDFS-15346.002.patch, > HDFS-15346.003.patch, HDFS-15346.004.patch, HDFS-15346.005.patch, > HDFS-15346.006.patch, HDFS-15346.007.patch > > > Patch in HDFS-15294 is too big to review so we split it into 2 patches. This > is the second one. Detail can be found at HDFS-15294. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15330) Document the ViewFSOverloadScheme details in ViewFS guide
[ https://issues.apache.org/jira/browse/HDFS-15330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15330: --- Status: Patch Available (was: In Progress) > Document the ViewFSOverloadScheme details in ViewFS guide > - > > Key: HDFS-15330 > URL: https://issues.apache.org/jira/browse/HDFS-15330 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > This Jira to track for documentation of ViewFSOverloadScheme usage guide. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15330) Document the ViewFSOverloadScheme details in ViewFS guide
[ https://issues.apache.org/jira/browse/HDFS-15330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126242#comment-17126242 ] Uma Maheswara Rao G commented on HDFS-15330: [https://github.com/umamaheswararao/hadoop/blob/HDFS-15330/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md] Image can't be rendered here from src. Here is the image link: [https://github.com/umamaheswararao/hadoop/blob/HDFS-15330/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/ViewFSOverloadScheme.png] Updated the PR for review! > Document the ViewFSOverloadScheme details in ViewFS guide > - > > Key: HDFS-15330 > URL: https://issues.apache.org/jira/browse/HDFS-15330 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > This Jira to track for documentation of ViewFSOverloadScheme usage guide. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15321) Make DFSAdmin tool to work with ViewFSOverloadScheme
[
https://issues.apache.org/jira/browse/HDFS-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126178#comment-17126178
]
Ayush Saxena edited comment on HDFS-15321 at 6/4/20, 8:59 PM:
--
A {{FileSystem.closeAll();}} in the {{tearDown()}} of the
{{TestDFSAdminWithHA}} fixes the test for me.
But we should fix this in DFSAdmin itself.
The reason of failure I feels now is : now {{FileSystem}} is not being stored
as part of {{FsShell#fs}}
So, when close is called for DFSAdmin, The close method of parent {{FsShell}}
is called which tends to close the {{FileSystem}}, since {{fs}} is not stored
now, the FileSystem doesn't get closed.
We should make sure the {{FileSystem}} is closed when {{DFSAdmin}} is closed.
I think we can even store it also like previously rather than everytime calling
{{AdminHelper.getDFS(getConf())}} whenever {{getDFS()}} is called, if there is
no specific reason for eliminating it now
was (Author: ayushtkn):
A {{FileSystem.closeAll();}} in the {{tearDown()}} of the
{{TestDFSAdminWithHA}} fixes the test for me.
> Make DFSAdmin tool to work with ViewFSOverloadScheme
>
>
> Key: HDFS-15321
> URL: https://issues.apache.org/jira/browse/HDFS-15321
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfsadmin, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> When we enable ViewFSOverLoadScheme and used hdfs scheme as overloaded
> scheme, users work with hdfs uris. But here DFSAdmin expects the impl classe
> to be DistribbuteFileSystem. If impl class is ViewFSoverloadScheme, it will
> fail.
> So, when impl is ViewFSoverloadScheme, we should get corresponding child hdfs
> to make DFSAdmin to work.
> This Jira makes the DFSAdmin to work with ViewFSoverloadScheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15321) Make DFSAdmin tool to work with ViewFSOverloadScheme
[
https://issues.apache.org/jira/browse/HDFS-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126178#comment-17126178
]
Ayush Saxena commented on HDFS-15321:
-
A {{FileSystem.closeAll();}} in the {{tearDown()}} of the
{{TestDFSAdminWithHA}} fixes the test for me.
> Make DFSAdmin tool to work with ViewFSOverloadScheme
>
>
> Key: HDFS-15321
> URL: https://issues.apache.org/jira/browse/HDFS-15321
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfsadmin, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> When we enable ViewFSOverLoadScheme and used hdfs scheme as overloaded
> scheme, users work with hdfs uris. But here DFSAdmin expects the impl classe
> to be DistribbuteFileSystem. If impl class is ViewFSoverloadScheme, it will
> fail.
> So, when impl is ViewFSoverloadScheme, we should get corresponding child hdfs
> to make DFSAdmin to work.
> This Jira makes the DFSAdmin to work with ViewFSoverloadScheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15198) RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode
[
https://issues.apache.org/jira/browse/HDFS-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126095#comment-17126095
]
Ayush Saxena commented on HDFS-15198:
-
Thanx [~zhengchenyu] for the patch.
Why you need to change {{FederationTestUtils}}, is the time limit creating
problem for this test?
> RBF: Add test for MountTableRefresherService failed to refresh other router
> MountTableEntries in secure mode
>
>
> Key: HDFS-15198
> URL: https://issues.apache.org/jira/browse/HDFS-15198
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Attachments: HDFS-15198.001.patch, HDFS-15198.002.patch,
> HDFS-15198.003.patch, HDFS-15198.004.patch, HDFS-15198.005.patch,
> HDFS-15198.006.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> In issue HDFS-13443, update mount table cache imediately. The specified
> router update their own mount table cache imediately, then update other's by
> rpc protocol refreshMountTableEntries. But in secure mode, can't refresh
> other's router's. In specified router's log, error like this
> {code}
> 2020-02-27 22:59:07,212 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 2020-02-27 22:59:07,213 ERROR
> org.apache.hadoop.hdfs.server.federation.router.MountTableRefresherThread:
> Failed to refresh mount table entries cache at router $host:8111
> java.io.IOException: DestHost:destPort host:8111 , LocalHost:localPort
> $host/$ip:0. Failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocolTranslatorPB.refreshMountTableEntries(RouterAdminProtocolTranslatorPB.java:288)
> at
> org.apache.hadoop.hdfs.server.federation.router.MountTableRefresherThread.run(MountTableRefresherThread.java:65)
> 2020-02-27 22:59:07,214 INFO
> org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver: Added
> new mount point /test_11 to resolver
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15376) Update the error about command line POST in httpfs documentation
[
https://issues.apache.org/jira/browse/HDFS-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126094#comment-17126094
]
Ayush Saxena commented on HDFS-15376:
-
+1
> Update the error about command line POST in httpfs documentation
>
>
> Key: HDFS-15376
> URL: https://issues.apache.org/jira/browse/HDFS-15376
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: httpfs
>Affects Versions: 3.2.1
>Reporter: bianqi
>Assignee: bianqi
>Priority: Major
> Attachments: HDFS-15376.001.patch
>
>
> In the official Hadoop documentation, there is an exception when executing
> the following command.
> {quote} {{curl -X POST
> 'http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=MKDIRS=foo'}}
> creates the HDFS {{/user/foo/bar}} directory.
> {quote}
> Command line returns results：
> {quote} *{"RemoteException":{"message":"Invalid HTTP POST operation
> [MKDIRS]","exception":"IOException","javaClassName":"java.io.IOException"}}*
> {quote}
>
> I checked the source code and found that the way to create the file should
> use PUT to submit the form.
> I modified to execute the command in PUT mode and got the result as
> follows
> {quote} {{curl -X PUT
> 'http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=MKDIRS=foo'}}
> creates the HDFS {{/user/foo/bar}} directory.
> {quote}
> Command line returns results：
> {"boolean":true}
> . At the same time the folder is created successfully.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15372) Files in snapshots no longer see attribute provider permissions
[
https://issues.apache.org/jira/browse/HDFS-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126044#comment-17126044
]
Stephen O'Donnell commented on HDFS-15372:
--
This patch makes things behave the same as they did on branch-2 with one
exception.
When checking a path like /data/.snapshot/snap1 the provider will see
/data/snap1, but on the branch-2, it would have seen /data/.snapshot/snap1.
With this patch and on branch-2 a path like /data/.snapshot/snap1/tab1 is seen
as /data/tab1 in the provider.
So far, I have not been able to figure out how it makes that distinction.
However from the code it is clear that the ".snapshot" part of the path is not
a real inode, and it only even gets a dummy permission / acl object returned,
so there must be some special handling for it somewhere.
> Files in snapshots no longer see attribute provider permissions
> ---
>
> Key: HDFS-15372
> URL: https://issues.apache.org/jira/browse/HDFS-15372
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-15372.001.patch
>
>
> Given a cluster with an authorization provider configured (eg Sentry) and the
> paths covered by the provider are snapshotable, there was a change in
> behaviour in how the provider permissions and ACLs are applied to files in
> snapshots between the 2.x branch and Hadoop 3.0.
> Eg, if we have the snapshotable path /data, which is Sentry managed. The ACLs
> below are provided by Sentry:
> {code}
> hadoop fs -getfacl -R /data
> # file: /data
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> # file: /data/tab1
> # owner: hive
> # group: hive
> user::rwx
> group::---
> group:flume:rwx
> user:hive:rwx
> group:hive:rwx
> group:testgroup:rwx
> mask::rwx
> other::--x
> /data/tab1
> {code}
> After taking a snapshot, the files in the snapshot do not see the provider
> permissions:
> {code}
> hadoop fs -getfacl -R /data/.snapshot
> # file: /data/.snapshot
> # owner:
> # group:
> user::rwx
> group::rwx
> other::rwx
> # file: /data/.snapshot/snap1
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> # file: /data/.snapshot/snap1/tab1
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> {code}
> However pre-Hadoop 3.0 (when the attribute provider etc was extensively
> refactored) snapshots did get the provider permissions.
> The reason is this code in FSDirectory.java which ultimately calls the
> attribute provider and passes the path we want permissions for:
> {code}
> INodeAttributes getAttributes(INodesInPath iip)
> throws IOException {
> INode node = FSDirectory.resolveLastINode(iip);
> int snapshot = iip.getPathSnapshotId();
> INodeAttributes nodeAttrs = node.getSnapshotINode(snapshot);
> UserGroupInformation ugi = NameNode.getRemoteUser();
> INodeAttributeProvider ap = this.getUserFilteredAttributeProvider(ugi);
> if (ap != null) {
> // permission checking sends the full components array including the
> // first empty component for the root. however file status
> // related calls are expected to strip out the root component according
> // to TestINodeAttributeProvider.
> byte[][] components = iip.getPathComponents();
> components = Arrays.copyOfRange(components, 1, components.length);
> nodeAttrs = ap.getAttributes(components, nodeAttrs);
> }
> return nodeAttrs;
> }
> {code}
> The line:
> {code}
> INode node = FSDirectory.resolveLastINode(iip);
> {code}
> Picks the last resolved Inode and if you then call node.getPathComponents,
> for a path like '/data/.snapshot/snap1/tab1' it will return /data/tab1. It
> resolves the snapshot path to its original location, but its still the
> snapshot inode.
> However the logic passes 'iip.getPathComponents' which returns
> "/user/.snapshot/snap1/tab" to the provider.
> The pre Hadoop 3.0 code passes the inode directly to the provider, and hence
> it only ever sees the path as "/user/data/tab1".
> It is debatable which path should be passed to the provider -
> /user/.snapshot/snap1/tab or /data/tab1 in the case of snapshots. However as
> the behaviour has changed I feel we should ensure the old behaviour is
> retained.
> It would also be fairly easy to provide a config switch so the provider gets
> the full snapshot path or the resolved path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15379) DataStreamer should reset thread interrupted state in createBlockOutputStream
[
https://issues.apache.org/jira/browse/HDFS-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125997#comment-17125997
]
Brahma Reddy Battula edited comment on HDFS-15379 at 6/4/20, 3:10 PM:
--
[~pilchard] thanks for reporting the issue and tagging me.
IMO, Channel operations are bound to the thread doing the operations. If this
thread is interrupted, the stream / channel is closed due to IO safety issues.
with this fix, I think, we are relaxing this??
was (Author: brahmareddy):
[~pilchard] thanks for reporting the issue and tagging me.
> DataStreamer should reset thread interrupted state in createBlockOutputStream
> -
>
> Key: HDFS-15379
> URL: https://issues.apache.org/jira/browse/HDFS-15379
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 2.7.7, 3.1.3
>Reporter: ludun
>Assignee: ludun
>Priority: Major
> Attachments: HDFS-15379.001.patch, HDFS-15379.002.patch,
> HDFS-15379.003.patch, HDFS-15379.004.patch
>
>
> In createBlockOutputStream if thread was interrupted becuase timeout to
> conenct to DataNode.
> {code}2020-05-27 18:32:53,310 | DEBUG | Connecting to datanode
> xx.xx.xx.xx:25009 | DataStreamer.java:251
> 2020-05-27 18:33:50,457 | INFO | Exception in createBlockOutputStream
> blk_1115121199_41386360 | DataStreamer.java:1854
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:40370
> remote=/xx.xx.xx.xx:25009]. 615000 millis timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:551)
> at
> org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1826)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1743)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> {code}
> then abandonBlockrpc to namenode also failed due to interrupted exception
> immediately.
> {code}2020-05-27 18:33:50,461 | DEBUG | Connecting to xx/xx.xx.xx.xx:25000 |
> Client.java:814
> 2020-05-27 18:33:50,462 | DEBUG | Failed to connect to server:
> xx/xx.xx.xx.xx:25000: try once and fail. | Client.java:956
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:720)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:823)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:436)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1613)
> at org.apache.hadoop.ipc.Client.call(Client.java:1444)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:234)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.abandonBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.abandonBlock(ClientNamenodeProtocolTranslatorPB.java:509)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
>
[jira] [Commented] (HDFS-15379) DataStreamer should reset thread interrupted state in createBlockOutputStream
[
https://issues.apache.org/jira/browse/HDFS-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125997#comment-17125997
]
Brahma Reddy Battula commented on HDFS-15379:
-
[~pilchard] thanks for reporting the issue and tagging me.
> DataStreamer should reset thread interrupted state in createBlockOutputStream
> -
>
> Key: HDFS-15379
> URL: https://issues.apache.org/jira/browse/HDFS-15379
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 2.7.7, 3.1.3
>Reporter: ludun
>Assignee: ludun
>Priority: Major
> Attachments: HDFS-15379.001.patch, HDFS-15379.002.patch,
> HDFS-15379.003.patch, HDFS-15379.004.patch
>
>
> In createBlockOutputStream if thread was interrupted becuase timeout to
> conenct to DataNode.
> {code}2020-05-27 18:32:53,310 | DEBUG | Connecting to datanode
> xx.xx.xx.xx:25009 | DataStreamer.java:251
> 2020-05-27 18:33:50,457 | INFO | Exception in createBlockOutputStream
> blk_1115121199_41386360 | DataStreamer.java:1854
> java.io.InterruptedIOException: Interrupted while waiting for IO on channel
> java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:40370
> remote=/xx.xx.xx.xx:25009]. 615000 millis timeout left.
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:551)
> at
> org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1826)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1743)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> {code}
> then abandonBlockrpc to namenode also failed due to interrupted exception
> immediately.
> {code}2020-05-27 18:33:50,461 | DEBUG | Connecting to xx/xx.xx.xx.xx:25000 |
> Client.java:814
> 2020-05-27 18:33:50,462 | DEBUG | Failed to connect to server:
> xx/xx.xx.xx.xx:25000: try once and fail. | Client.java:956
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:720)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:823)
> at org.apache.hadoop.ipc.Client$Connection.access$3700(Client.java:436)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1613)
> at org.apache.hadoop.ipc.Client.call(Client.java:1444)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:234)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.abandonBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.abandonBlock(ClientNamenodeProtocolTranslatorPB.java:509)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy11.abandonBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1748)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:718)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HDFS-15372) Files in snapshots no longer see attribute provider permissions
[
https://issues.apache.org/jira/browse/HDFS-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125979#comment-17125979
]
Stephen O'Donnell commented on HDFS-15372:
--
Consider a path like "/data/.snapshot/snapshot_1/tab1".
With the 001 patch in place, if you try to list /data/.snapshot/snapshot_1, the
path seen by the attribute provider is:
/user/snapshot_1
Before, it was:
/user/.snapshot/snapshot1
If you then try to list something inside the snapshot, eg
/user/.snapshot/snapshot_1/tab1, the provider sees:
/user/tab1
Previously, this was:
/user/.snapshot/snapshot_1/tab1
I need to try to confirm what the old behaviour was on branch 2, and if it was
similar.
> Files in snapshots no longer see attribute provider permissions
> ---
>
> Key: HDFS-15372
> URL: https://issues.apache.org/jira/browse/HDFS-15372
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-15372.001.patch
>
>
> Given a cluster with an authorization provider configured (eg Sentry) and the
> paths covered by the provider are snapshotable, there was a change in
> behaviour in how the provider permissions and ACLs are applied to files in
> snapshots between the 2.x branch and Hadoop 3.0.
> Eg, if we have the snapshotable path /data, which is Sentry managed. The ACLs
> below are provided by Sentry:
> {code}
> hadoop fs -getfacl -R /data
> # file: /data
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> # file: /data/tab1
> # owner: hive
> # group: hive
> user::rwx
> group::---
> group:flume:rwx
> user:hive:rwx
> group:hive:rwx
> group:testgroup:rwx
> mask::rwx
> other::--x
> /data/tab1
> {code}
> After taking a snapshot, the files in the snapshot do not see the provider
> permissions:
> {code}
> hadoop fs -getfacl -R /data/.snapshot
> # file: /data/.snapshot
> # owner:
> # group:
> user::rwx
> group::rwx
> other::rwx
> # file: /data/.snapshot/snap1
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> # file: /data/.snapshot/snap1/tab1
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> {code}
> However pre-Hadoop 3.0 (when the attribute provider etc was extensively
> refactored) snapshots did get the provider permissions.
> The reason is this code in FSDirectory.java which ultimately calls the
> attribute provider and passes the path we want permissions for:
> {code}
> INodeAttributes getAttributes(INodesInPath iip)
> throws IOException {
> INode node = FSDirectory.resolveLastINode(iip);
> int snapshot = iip.getPathSnapshotId();
> INodeAttributes nodeAttrs = node.getSnapshotINode(snapshot);
> UserGroupInformation ugi = NameNode.getRemoteUser();
> INodeAttributeProvider ap = this.getUserFilteredAttributeProvider(ugi);
> if (ap != null) {
> // permission checking sends the full components array including the
> // first empty component for the root. however file status
> // related calls are expected to strip out the root component according
> // to TestINodeAttributeProvider.
> byte[][] components = iip.getPathComponents();
> components = Arrays.copyOfRange(components, 1, components.length);
> nodeAttrs = ap.getAttributes(components, nodeAttrs);
> }
> return nodeAttrs;
> }
> {code}
> The line:
> {code}
> INode node = FSDirectory.resolveLastINode(iip);
> {code}
> Picks the last resolved Inode and if you then call node.getPathComponents,
> for a path like '/data/.snapshot/snap1/tab1' it will return /data/tab1. It
> resolves the snapshot path to its original location, but its still the
> snapshot inode.
> However the logic passes 'iip.getPathComponents' which returns
> "/user/.snapshot/snap1/tab" to the provider.
> The pre Hadoop 3.0 code passes the inode directly to the provider, and hence
> it only ever sees the path as "/user/data/tab1".
> It is debatable which path should be passed to the provider -
> /user/.snapshot/snap1/tab or /data/tab1 in the case of snapshots. However as
> the behaviour has changed I feel we should ensure the old behaviour is
> retained.
> It would also be fairly easy to provide a config switch so the provider gets
> the full snapshot path or the resolved path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15346) RBF: DistCpFedBalance implementation
[ https://issues.apache.org/jira/browse/HDFS-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125790#comment-17125790 ] Jinglun commented on HDFS-15346: Hi [~linyiqun], v07 has passed the tests. Could you help to review it. Thanks very much ! > RBF: DistCpFedBalance implementation > > > Key: HDFS-15346 > URL: https://issues.apache.org/jira/browse/HDFS-15346 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15346.001.patch, HDFS-15346.002.patch, > HDFS-15346.003.patch, HDFS-15346.004.patch, HDFS-15346.005.patch, > HDFS-15346.006.patch, HDFS-15346.007.patch > > > Patch in HDFS-15294 is too big to review so we split it into 2 patches. This > is the second one. Detail can be found at HDFS-15294. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12571) Ozone: remove spaces from the beginning of the hdfs script
[
https://issues.apache.org/jira/browse/HDFS-12571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125635#comment-17125635
]
Jialin Liu commented on HDFS-12571:
---
as of June 14 2020, I'm still seeing the same issue:
{code:java}
sudo bash start-all.sh
Password:
Starting namenodes on [localhost]
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh:
line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh:
line 398: ` done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
249: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
257: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 218:
hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 219:
hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
268: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
275: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
278: hadoop_generic_java_subcmd_handler: command not found
Starting datanodes
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh:
line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh:
line 398: ` done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
249: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
257: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 218:
hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 219:
hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
268: hadoop_add_client_opts: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
275: hadoop_subcommand_opts: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
278: hadoop_generic_java_subcmd_handler: command not found
Starting secondary namenodes [Jialin.local]
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh:
line 398: syntax error near unexpected token `<'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-functions.sh:
line 398: ` done < <(for text in "${input[@]}"; do'
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
70: hadoop_deprecate_envvar: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
87: hadoop_bootstrap: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
104: hadoop_parse_args: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
105: shift: : numeric argument required
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
249: hadoop_need_reexec: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
257: hadoop_verify_user_perm: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 218:
hadoop_validate_classname: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/hdfs: line 219:
hadoop_exit_with_usage: command not found
/usr/local/Cellar/hadoop/3.2.1_1/libexec/bin/../libexec/hadoop-config.sh: line
268: hadoop_add_client_opts: command not found
[jira] [Commented] (HDFS-15321) Make DFSAdmin tool to work with ViewFSOverloadScheme
[
https://issues.apache.org/jira/browse/HDFS-15321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125609#comment-17125609
]
Ayush Saxena commented on HDFS-15321:
-
Hi [~umamaheswararao] seems this change broke {{TestDFSAdminWithHA}}
The test failed in both Jenkins report of the PR too :
https://builds.apache.org/job/hadoop-multibranch/job/PR-2041/1/testReport/
https://builds.apache.org/job/hadoop-multibranch/job/PR-2041/2/testReport/
Can you check once.
> Make DFSAdmin tool to work with ViewFSOverloadScheme
>
>
> Key: HDFS-15321
> URL: https://issues.apache.org/jira/browse/HDFS-15321
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfsadmin, fs, viewfs
>Affects Versions: 3.2.1
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> When we enable ViewFSOverLoadScheme and used hdfs scheme as overloaded
> scheme, users work with hdfs uris. But here DFSAdmin expects the impl classe
> to be DistribbuteFileSystem. If impl class is ViewFSoverloadScheme, it will
> fail.
> So, when impl is ViewFSoverloadScheme, we should get corresponding child hdfs
> to make DFSAdmin to work.
> This Jira makes the DFSAdmin to work with ViewFSoverloadScheme.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
