[jira] [Resolved] (HADOOP-18128) outputstream.md typo issue
[ https://issues.apache.org/jira/browse/HADOOP-18128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HADOOP-18128. -- Fix Version/s: 3.4.0 Resolution: Fixed > outputstream.md typo issue > -- > > Key: HADOOP-18128 > URL: https://issues.apache.org/jira/browse/HADOOP-18128 > Project: Hadoop Common > Issue Type: Improvement >Reporter: leo sun >Assignee: leo sun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2022-02-17-10-53-01-704.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > There is a typo issue in outputstream.md on the branch – trunk > !image-2022-02-17-10-53-01-704.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18117) Add an option to preserve root directory permissions
[
https://issues.apache.org/jira/browse/HADOOP-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei resolved HADOOP-18117.
--
Fix Version/s: 3.4.0
Resolution: Fixed
> Add an option to preserve root directory permissions
>
>
> Key: HADOOP-18117
> URL: https://issues.apache.org/jira/browse/HADOOP-18117
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Mohanad Elsafty
>Assignee: Mohanad Elsafty
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> As mentioned in https://issues.apache.org/jira/browse/HADOOP-15211
>
> If *-update* or *-overwrite* is being passed when *distcp* used, the root
> directory will be skipped in two occasions (CopyListing#doBuildListing &
> CopyCommitter#preserveFileAttributesForDirectories), which will ignore root
> directory's attributes.
>
> We face the same issue when distcp huge data between clusters and it takes
> too much effort to update root directories attributes manually.
>
> From the earlier ticket it's obvious why this behaviour is there, but
> sometime we need to enforce root directory update hence I will add a new
> option for distcp to enable someone (who understands the need of this and
> know what they are doing) to enforce the update of root directory's
> attributes (permissions, ownership, ...)
>
> It should be simple one, something like this
> {code:java}
> $ hadoop distcp -p -update -updateRootDirectoryAttributes /a/b/c /a/b/d {code}
> This behaviour is optional and will be *false* by default. (it should not
> affect existing *distcp* users).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18060) RPCMetrics increases the number of handlers in processing
[ https://issues.apache.org/jira/browse/HADOOP-18060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HADOOP-18060. -- Fix Version/s: 3.4.0 Resolution: Fixed > RPCMetrics increases the number of handlers in processing > - > > Key: HADOOP-18060 > URL: https://issues.apache.org/jira/browse/HADOOP-18060 > Project: Hadoop Common > Issue Type: Improvement > Components: rpc-server >Affects Versions: 2.9.2 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > When using RPC, we recorded a lot of useful information, such as Queue time, > Processing time. These are very helpful. > But we can't know how many handlers are actually working now (only those that > handle Call), especially when the Call Queue is very high. This is also not > conducive to us optimizing the cluster. > It would be very helpful if we can see the number of handlers being processed > in RPCMetrics. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18057) Fix typo: validateEncrytionSecrets -> validateEncryptionSecrets
[
https://issues.apache.org/jira/browse/HADOOP-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei resolved HADOOP-18057.
--
Fix Version/s: 3.4.0
(was: 3.3.1)
Resolution: Fixed
> Fix typo: validateEncrytionSecrets -> validateEncryptionSecrets
> ---
>
> Key: HADOOP-18057
> URL: https://issues.apache.org/jira/browse/HADOOP-18057
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Fixing typo validateEncrytionSecrets to validateEncryptionSecrets in
> {code:java}
> hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractTestS3AEncryption.java{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17995) Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson
[ https://issues.apache.org/jira/browse/HADOOP-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HADOOP-17995. -- Resolution: Fixed > Stale record should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson > > > Key: HADOOP-17995 > URL: https://issues.apache.org/jira/browse/HADOOP-17995 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > As [HADOOP-16947|https://issues.apache.org/jira/browse/HADOOP-16947] problem > with description, > Stale SumAndCount also should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson. > Ensure the DataNode JMX get SendPacketDownstreamAvgInfo Metrics is accurate -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17995) Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson
[ https://issues.apache.org/jira/browse/HADOOP-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei reopened HADOOP-17995: -- > Stale record should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson > > > Key: HADOOP-17995 > URL: https://issues.apache.org/jira/browse/HADOOP-17995 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > As [HADOOP-16947|https://issues.apache.org/jira/browse/HADOOP-16947] problem > with description, > Stale SumAndCount also should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson. > Ensure the DataNode JMX get SendPacketDownstreamAvgInfo Metrics is accurate -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17995) Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson
[ https://issues.apache.org/jira/browse/HADOOP-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HADOOP-17995. -- Fix Version/s: 3.4.0 Resolution: Fixed > Stale record should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson > > > Key: HADOOP-17995 > URL: https://issues.apache.org/jira/browse/HADOOP-17995 > Project: Hadoop Common > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > As [HADOOP-16947|https://issues.apache.org/jira/browse/HADOOP-16947] problem > with description, > Stale SumAndCount also should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson. > Ensure the DataNode JMX get SendPacketDownstreamAvgInfo Metrics is accurate -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17914) Print RPC response length in the exception message
[ https://issues.apache.org/jira/browse/HADOOP-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HADOOP-17914. -- Fix Version/s: 3.4.0 Resolution: Fixed > Print RPC response length in the exception message > -- > > Key: HADOOP-17914 > URL: https://issues.apache.org/jira/browse/HADOOP-17914 > Project: Hadoop Common > Issue Type: Improvement >Reporter: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > To facilitate problem tracking, we can print RPC Response Length in the > exception message. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17846) Update link of wiki on README
[ https://issues.apache.org/jira/browse/HADOOP-17846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Fei resolved HADOOP-17846. -- Resolution: Not A Problem > Update link of wiki on README > - > > Key: HADOOP-17846 > URL: https://issues.apache.org/jira/browse/HADOOP-17846 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Yi-Sheng Lien >Assignee: Yi-Sheng Lien >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The [README|https://github.com/apache/hadoop#readme] on [github > repo|https://github.com/apache/hadoop] is deprecated now. > Let's update it to > [HADOOP2|https://cwiki.apache.org/confluence/display/HADOOP2] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17749) Remove lock contention in SelectorPool of SocketIOWithTimeout
[
https://issues.apache.org/jira/browse/HADOOP-17749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei resolved HADOOP-17749.
--
Resolution: Fixed
> Remove lock contention in SelectorPool of SocketIOWithTimeout
> -
>
> Key: HADOOP-17749
> URL: https://issues.apache.org/jira/browse/HADOOP-17749
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: Xuesen Liang
>Assignee: Xuesen Liang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> *SelectorPool* in
> hadoop-common/src/main/java/org/apache/hadoop/net/*SocketIOWithTimeout.java*
> is a point of lock contention.
> For example:
> {code:java}
> $ grep 'waiting to lock <0x7f7d94006d90>' 63692.jstack | uniq -c
> 1005 - waiting to lock <0x7f7d94006d90> (a
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
> {code}
> and the thread stack is as follows:
> {code:java}
> "IPC Client (324579982) connection to /100.10.6.10:60020 from user_00" #14139
> daemon prio=5 os_prio=0 tid=0x7f7374039000 nid=0x85cc waiting for monitor
> entry [0x7f6f45939000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:390)
> - waiting to lock <0x7f7d94006d90> (a
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> - locked <0x7fa818caf258> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.readResponse(RpcClientImpl.java:967)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568)
> {code}
> We should remove the lock contention.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17749) Remove lock contention in SelectorPool of SocketIOWithTimeout
[
https://issues.apache.org/jira/browse/HADOOP-17749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei reopened HADOOP-17749:
--
> Remove lock contention in SelectorPool of SocketIOWithTimeout
> -
>
> Key: HADOOP-17749
> URL: https://issues.apache.org/jira/browse/HADOOP-17749
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: Xuesen Liang
>Assignee: Xuesen Liang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> *SelectorPool* in
> hadoop-common/src/main/java/org/apache/hadoop/net/*SocketIOWithTimeout.java*
> is a point of lock contention.
> For example:
> {code:java}
> $ grep 'waiting to lock <0x7f7d94006d90>' 63692.jstack | uniq -c
> 1005 - waiting to lock <0x7f7d94006d90> (a
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
> {code}
> and the thread stack is as follows:
> {code:java}
> "IPC Client (324579982) connection to /100.10.6.10:60020 from user_00" #14139
> daemon prio=5 os_prio=0 tid=0x7f7374039000 nid=0x85cc waiting for monitor
> entry [0x7f6f45939000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:390)
> - waiting to lock <0x7f7d94006d90> (a
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> - locked <0x7fa818caf258> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.readResponse(RpcClientImpl.java:967)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568)
> {code}
> We should remove the lock contention.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17749) Remove lock contention in SelectorPool of SocketIOWithTimeout
[
https://issues.apache.org/jira/browse/HADOOP-17749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei resolved HADOOP-17749.
--
Fix Version/s: 3.4.0
Resolution: Fixed
> Remove lock contention in SelectorPool of SocketIOWithTimeout
> -
>
> Key: HADOOP-17749
> URL: https://issues.apache.org/jira/browse/HADOOP-17749
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: Xuesen Liang
>Assignee: Xuesen Liang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> *SelectorPool* in
> hadoop-common/src/main/java/org/apache/hadoop/net/*SocketIOWithTimeout.java*
> is a point of lock contention.
> For example:
> {code:java}
> $ grep 'waiting to lock <0x7f7d94006d90>' 63692.jstack | uniq -c
> 1005 - waiting to lock <0x7f7d94006d90> (a
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
> {code}
> and the thread stack is as follows:
> {code:java}
> "IPC Client (324579982) connection to /100.10.6.10:60020 from user_00" #14139
> daemon prio=5 os_prio=0 tid=0x7f7374039000 nid=0x85cc waiting for monitor
> entry [0x7f6f45939000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:390)
> - waiting to lock <0x7f7d94006d90> (a
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool)
> at
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> - locked <0x7fa818caf258> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.readResponse(RpcClientImpl.java:967)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568)
> {code}
> We should remove the lock contention.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
