[jira] [Commented] (HADOOP-19186) Change loglevel to ERROR/WARNING so that it would easy to identify the problem without ignoring
[ https://issues.apache.org/jira/browse/HADOOP-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17852491#comment-17852491 ] Arpit Agarwal commented on HADOOP-19186: Hi [~smajeti], this is most likely being logged at DEBUG to avoid spamming the service logs. > Change loglevel to ERROR/WARNING so that it would easy to identify the > problem without ignoring > --- > > Key: HADOOP-19186 > URL: https://issues.apache.org/jira/browse/HADOOP-19186 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Srinivasu Majeti >Priority: Major > > On the new Host with Java version 11, the DN was not able to communicate with > the NN. We enabled DEBUG logging for the DN and the below message was logged > under DEBUG level. > DEBUG org.apache.hadoop.security.UserGroupInformation: > PrivilegedActionException > as:hdfs/av3l704p.bigdata.it.internal@PRODUCTION.LOCAL (auth:KERBEROS) > cause:javax.security.sasl.SaslExcept > ion: GSS initiate failed [Caused by GSSException: No valid credentials > provided (Mechanism level: Receive timed out)] > Without a DEBUG level logging, this was shown up as a WARNING as below > WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for > hdfs/av3l704p.bigdata.it.internal@PRODUCTION.LOCAL to > avl2785p.bigdata.it.internal/172.24.178.32:8022 > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Receive timed > out)] > A considerable amount of time was spent troubleshooting this issue as this > exception was moved to a DEBUG level which was difficult to track in the logs. > Can we have such critical WARNINGs shown up at the WARN/ERROR level so that > it's not missed when we enable DEBUG level logging for datanodes? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18851) AbstractDelegationTokenSecretManager- Performance improvement by optimising the synchronization context
[
https://issues.apache.org/jira/browse/HADOOP-18851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal reassigned HADOOP-18851:
--
Assignee: Vikas Kumar
> AbstractDelegationTokenSecretManager- Performance improvement by optimising
> the synchronization context
> ---
>
> Key: HADOOP-18851
> URL: https://issues.apache.org/jira/browse/HADOOP-18851
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Vikas Kumar
>Assignee: Vikas Kumar
>Priority: Major
> Attachments: Screenshot 2023-08-16 at 5.36.57 PM.png
>
>
> *Context:*
> KMS depends on hadoop-common for DT management. Recently we were analysing
> one performance issue and following is out findings:
> # Around 96% (196 out of 200) KMS container threads were in BLOCKED state at
> following:
> ## *AbstractDelegationTokenSecretManager.verifyToken()*
> ## *AbstractDelegationTokenSecretManager.createPassword()*
> # And then process crashed.
>
> {code:java}
> http-nio-9292-exec-200PRIORITY : 5THREAD ID : 0X7F075C157800NATIVE ID :
> 0X2C87FNATIVE ID (DECIMAL) : 182399STATE : BLOCKED
> stackTrace:
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.verifyToken(AbstractDelegationTokenSecretManager.java:474)
> - waiting to lock <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.verifyToken(DelegationTokenManager.java:213)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:396)
> at {code}
> All the 199 out of 200 were blocked at above point.
> And the lock they are waiting for is acquired by a thread that was trying to
> createPassword and publishing the same on ZK.
>
> {code:java}
> stackTrace:
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1598)
> - locked <0x000749263ec0> (a org.apache.zookeeper.ClientCnxn$Packet)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1570)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2235)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:398)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl$7.call(SetDataBuilderImpl.java:385)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:382)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:358)
> at
> org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:36)
> at
> org.apache.curator.framework.recipes.shared.SharedValue.trySetValue(SharedValue.java:201)
> at
> org.apache.curator.framework.recipes.shared.SharedCount.trySetCount(SharedCount.java:116)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrSharedCount(ZKDelegationTokenSecretManager.java:586)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:601)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:402)
> - locked <0x0005f2f545e8> (a
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:48)
> at org.apache.hadoop.security.token.Token.(Token.java:67)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.createToken(DelegationTokenManager.java:183)
> {code}
> We can say that this thread is slow and has blocked remaining all. But
> following is my observation:
>
> # verifyToken() and createPaswword() has been synchronized because one is
> reading the tokenMap and another is updating the map. If it's only to protect
> the map, then can't we simply use ConcurrentHashMap and remove the
> "synchronized" keyword. Because due to this, all reader threads ( to
> verifyToken()) are also blocking each other.
> # IN HA env, It is recommended to use ZK to store DTs. We know that
> CuratorFramework is thread safe.
> ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum() only requires
> to be protected from concurrent execution and it should
[jira] [Assigned] (HADOOP-18514) Remove the legacy Ozone website
[ https://issues.apache.org/jira/browse/HADOOP-18514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HADOOP-18514: -- Assignee: Siddharth Wagle > Remove the legacy Ozone website > --- > > Key: HADOOP-18514 > URL: https://issues.apache.org/jira/browse/HADOOP-18514 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Arpit Agarwal >Assignee: Siddharth Wagle >Priority: Major > > Let's remove the old Ozone website: https://hadoop.apache.org/ozone/ > Since Ozone has moved to a separate TLP long ago with its own website. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18514) Remove the legacy Ozone website
Arpit Agarwal created HADOOP-18514: -- Summary: Remove the legacy Ozone website Key: HADOOP-18514 URL: https://issues.apache.org/jira/browse/HADOOP-18514 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Arpit Agarwal Let's remove the old Ozone website: https://hadoop.apache.org/ozone/ Since Ozone has moved to a separate TLP long ago with its own website. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18306) Warnings should not be shown on cli console when linux user not present on client
[
https://issues.apache.org/jira/browse/HADOOP-18306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17565941#comment-17565941
]
Arpit Agarwal commented on HADOOP-18306:
Added you as a Hadoop contributor and assigned to you. Thanks for the
contribution [~swamirishi].
> Warnings should not be shown on cli console when linux user not present on
> client
> -
>
> Key: HADOOP-18306
> URL: https://issues.apache.org/jira/browse/HADOOP-18306
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Swaminathan Balachandran
>Assignee: Swaminathan Balachandran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {noformat}
> 22/04/26 10:51:18 WARN security.ShellBasedUnixGroupsMapping: unable to return
> groups for user om
> PartialGroupNameException The user name 'om' is not found. id: ‘om’: no such
> user
> id: ‘om’: no such user
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:291)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:215)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroupsSet(ShellBasedUnixGroupsMapping.java:123)
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupSet(Groups.java:413)
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:351)
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:300)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3962)
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3985)
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4946)
> at org.apache.hadoop.security.Groups.getGroupInternal(Groups.java:258)
> at org.apache.hadoop.security.Groups.getGroupsSet(Groups.java:230)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupsSet(UserGroupInformation.java:1760)
> at
> org.apache.hadoop.security.UserGroupInformation.getPrimaryGroupName(UserGroupInformation.java:1603)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.getGroupName(BasicRootedOzoneClientAdapterImpl.java:992)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.getFileStatusAdapterForVolume(BasicRootedOzoneClientAdapterImpl.java:1016)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.getFileStatus(BasicRootedOzoneClientAdapterImpl.java:574)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.getFileStatus(BasicRootedOzoneFileSystem.java:858)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2106)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.globStatus(BasicRootedOzoneFileSystem.java:915)
> at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:353)
> at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250)
> at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.ozone.OzoneFsShell.main(OzoneFsShell.java:83)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18306) Warnings should not be shown on cli console when linux user not present on client
[
https://issues.apache.org/jira/browse/HADOOP-18306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal reassigned HADOOP-18306:
--
Assignee: Swaminathan Balachandran
> Warnings should not be shown on cli console when linux user not present on
> client
> -
>
> Key: HADOOP-18306
> URL: https://issues.apache.org/jira/browse/HADOOP-18306
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Swaminathan Balachandran
>Assignee: Swaminathan Balachandran
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {noformat}
> 22/04/26 10:51:18 WARN security.ShellBasedUnixGroupsMapping: unable to return
> groups for user om
> PartialGroupNameException The user name 'om' is not found. id: ‘om’: no such
> user
> id: ‘om’: no such user
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.resolvePartialGroupNames(ShellBasedUnixGroupsMapping.java:291)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getUnixGroups(ShellBasedUnixGroupsMapping.java:215)
> at
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping.getGroupsSet(ShellBasedUnixGroupsMapping.java:123)
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.fetchGroupSet(Groups.java:413)
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:351)
> at
> org.apache.hadoop.security.Groups$GroupCacheLoader.load(Groups.java:300)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
> at com.google.common.cache.LocalCache.get(LocalCache.java:3962)
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3985)
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4946)
> at org.apache.hadoop.security.Groups.getGroupInternal(Groups.java:258)
> at org.apache.hadoop.security.Groups.getGroupsSet(Groups.java:230)
> at
> org.apache.hadoop.security.UserGroupInformation.getGroupsSet(UserGroupInformation.java:1760)
> at
> org.apache.hadoop.security.UserGroupInformation.getPrimaryGroupName(UserGroupInformation.java:1603)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.getGroupName(BasicRootedOzoneClientAdapterImpl.java:992)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.getFileStatusAdapterForVolume(BasicRootedOzoneClientAdapterImpl.java:1016)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.getFileStatus(BasicRootedOzoneClientAdapterImpl.java:574)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.getFileStatus(BasicRootedOzoneFileSystem.java:858)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:2106)
> at
> org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.globStatus(BasicRootedOzoneFileSystem.java:915)
> at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:353)
> at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250)
> at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.ozone.OzoneFsShell.main(OzoneFsShell.java:83)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18120) Hadoop auth does not handle HTTP Headers in a case-insensitive way
[ https://issues.apache.org/jira/browse/HADOOP-18120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HADOOP-18120: -- Assignee: Daniel Fritsi > Hadoop auth does not handle HTTP Headers in a case-insensitive way > -- > > Key: HADOOP-18120 > URL: https://issues.apache.org/jira/browse/HADOOP-18120 > Project: Hadoop Common > Issue Type: Bug > Components: auth >Reporter: Daniel Fritsi >Assignee: Daniel Fritsi >Priority: Critical > Attachments: HADOOP-18120.patch > > > According to [RFC-2616|https://www.ietf.org/rfc/rfc2616.txt] HTTP Headers are > case-insensitive. There are proxies / load balancers (e.g.: newer versions of > HA-proxy) which deliberately make some of the HTTP headers lower-case results > in an authentication / authorization failure inside the Hadoop codebase. > I've created a small patch (I'm from Cloudera): > [^hadoop-auth-headers.patch]. This resolves our authentication issue. Can > someone please have a look at this? -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[ https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379920#comment-17379920 ] Arpit Agarwal commented on HADOOP-11890: Agreed that 1 is better, to keep the individual commit history. > Uber-JIRA: Hadoop should support IPv6 > - > > Key: HADOOP-11890 > URL: https://issues.apache.org/jira/browse/HADOOP-11890 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Major > Labels: ipv6 > Attachments: hadoop_2.7.3_ipv6_commits.txt > > > Hadoop currently treats IPv6 as unsupported. Track related smaller issues to > support IPv6. > (Current case here is mainly HBase on HDFS, so any suggestions about other > test cases/workload are really appreciated.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[ https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17377567#comment-17377567 ] Arpit Agarwal commented on HADOOP-11890: bq. Now we come with an approach, Under Trunk , Create a new branch for IPV6 and rebase the all Jiras committed on branch HADOOP-11890 to new IPV6 branch Hi [~hemanthboyina], that makes sense. You may also want to start a branch merge discussion on the dev mailing list when you're ready. > Uber-JIRA: Hadoop should support IPv6 > - > > Key: HADOOP-11890 > URL: https://issues.apache.org/jira/browse/HADOOP-11890 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Major > Labels: ipv6 > Attachments: hadoop_2.7.3_ipv6_commits.txt > > > Hadoop currently treats IPv6 as unsupported. Track related smaller issues to > support IPv6. > (Current case here is mainly HBase on HDFS, so any suggestions about other > test cases/workload are really appreciated.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[ https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17363662#comment-17363662 ] Arpit Agarwal commented on HADOOP-11890: Thanks [~hemanthboyina]. What do you see as the next steps? Do you want to propose merging your rebased changes on Apache trunk? > Uber-JIRA: Hadoop should support IPv6 > - > > Key: HADOOP-11890 > URL: https://issues.apache.org/jira/browse/HADOOP-11890 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Major > Labels: ipv6 > Attachments: hadoop_2.7.3_ipv6_commits.txt > > > Hadoop currently treats IPv6 as unsupported. Track related smaller issues to > support IPv6. > (Current case here is mainly HBase on HDFS, so any suggestions about other > test cases/workload are really appreciated.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11890) Uber-JIRA: Hadoop should support IPv6
[ https://issues.apache.org/jira/browse/HADOOP-11890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356649#comment-17356649 ] Arpit Agarwal commented on HADOOP-11890: [~hemanthboyina], [~brahmareddy], did you ever get to trying out these changes? What were your findings? > Uber-JIRA: Hadoop should support IPv6 > - > > Key: HADOOP-11890 > URL: https://issues.apache.org/jira/browse/HADOOP-11890 > Project: Hadoop Common > Issue Type: Improvement > Components: net >Reporter: Nate Edel >Assignee: Nate Edel >Priority: Major > Labels: ipv6 > Attachments: hadoop_2.7.3_ipv6_commits.txt > > > Hadoop currently treats IPv6 as unsupported. Track related smaller issues to > support IPv6. > (Current case here is mainly HBase on HDFS, so any suggestions about other > test cases/workload are really appreciated.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15327) Upgrade MR ShuffleHandler to use Netty4
[ https://issues.apache.org/jira/browse/HADOOP-15327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HADOOP-15327: -- Assignee: (was: Wei-Chiu Chuang) > Upgrade MR ShuffleHandler to use Netty4 > --- > > Key: HADOOP-15327 > URL: https://issues.apache.org/jira/browse/HADOOP-15327 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Priority: Major > > This way, we can remove the dependencies on the netty3 (jboss.netty) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17169) Remove whitelist/blacklist terminology from Hadoop Common
[
https://issues.apache.org/jira/browse/HADOOP-17169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199582#comment-17199582
]
Arpit Agarwal commented on HADOOP-17169:
{quote}At what point is it acceptable to remove them?
{quote}
At the next major release where we don't give any guarantee of compatibility? I
see you started an email thread. I will also respond there, perhaps others can
chime in too. Thanks for starting this important discussion.
> Remove whitelist/blacklist terminology from Hadoop Common
> -
>
> Key: HADOOP-17169
> URL: https://issues.apache.org/jira/browse/HADOOP-17169
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: HADOOP-17169.001.patch, HADOOP-17169.002.patch,
> HADOOP-17169.003.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17169) Remove whitelist/blacklist terminology from Hadoop Common
[ https://issues.apache.org/jira/browse/HADOOP-17169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199568#comment-17199568 ] Arpit Agarwal commented on HADOOP-17169: I don't think we should go overboard and remove the old config key names. This will be highly disruptive for future release adoption. We can mark them as deprecated though, and consider removing them in a subsequent major release. > Remove whitelist/blacklist terminology from Hadoop Common > - > > Key: HADOOP-17169 > URL: https://issues.apache.org/jira/browse/HADOOP-17169 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: HADOOP-17169.001.patch, HADOOP-17169.002.patch, > HADOOP-17169.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17116) Skip Retry INFO logging on first failover from a proxy
[ https://issues.apache.org/jira/browse/HADOOP-17116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-17116: --- Component/s: ha +1 pending Jenkins. Thanks for reporting this and the fix [~hanishakoneru]. > Skip Retry INFO logging on first failover from a proxy > -- > > Key: HADOOP-17116 > URL: https://issues.apache.org/jira/browse/HADOOP-17116 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HADOOP-17116.001.patch, HADOOP-17116.002.patch, > HADOOP-17116.003.patch > > > RetryInvocationHandler logs an INFO level message on every failover except > the first. This used to be ideal before when there were only 2 proxies in the > FailoverProxyProvider. But if there are more than 2 proxies (as is possible > with 3 or more NNs in HA), then there could be more than one failover to find > the currently active proxy. > To avoid creating noise in clients logs/ console, RetryInvocationHandler > should skip logging once for each proxy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17116) Skip Retry INFO logging on first failover from a proxy
[ https://issues.apache.org/jira/browse/HADOOP-17116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-17116: --- Priority: Blocker (was: Major) > Skip Retry INFO logging on first failover from a proxy > -- > > Key: HADOOP-17116 > URL: https://issues.apache.org/jira/browse/HADOOP-17116 > Project: Hadoop Common > Issue Type: Task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Blocker > Attachments: HADOOP-17116.001.patch > > > RetryInvocationHandler logs an INFO level message on every failover except > the first. This used to be ideal before when there were only 2 proxies in the > FailoverProxyProvider. But if there are more than 2 proxies (as is possible > with 3 or more NNs in HA), then there could be more than one failover to find > the currently active proxy. > To avoid creating noise in clients logs/ console, RetryInvocationHandler > should skip logging once for each proxy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17116) Skip Retry INFO logging on first failover from a proxy
[ https://issues.apache.org/jira/browse/HADOOP-17116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-17116: --- Target Version/s: 3.3.1 > Skip Retry INFO logging on first failover from a proxy > -- > > Key: HADOOP-17116 > URL: https://issues.apache.org/jira/browse/HADOOP-17116 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HADOOP-17116.001.patch > > > RetryInvocationHandler logs an INFO level message on every failover except > the first. This used to be ideal before when there were only 2 proxies in the > FailoverProxyProvider. But if there are more than 2 proxies (as is possible > with 3 or more NNs in HA), then there could be more than one failover to find > the currently active proxy. > To avoid creating noise in clients logs/ console, RetryInvocationHandler > should skip logging once for each proxy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17116) Skip Retry INFO logging on first failover from a proxy
[ https://issues.apache.org/jira/browse/HADOOP-17116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-17116: --- Priority: Major (was: Blocker) > Skip Retry INFO logging on first failover from a proxy > -- > > Key: HADOOP-17116 > URL: https://issues.apache.org/jira/browse/HADOOP-17116 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HADOOP-17116.001.patch > > > RetryInvocationHandler logs an INFO level message on every failover except > the first. This used to be ideal before when there were only 2 proxies in the > FailoverProxyProvider. But if there are more than 2 proxies (as is possible > with 3 or more NNs in HA), then there could be more than one failover to find > the currently active proxy. > To avoid creating noise in clients logs/ console, RetryInvocationHandler > should skip logging once for each proxy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17116) Skip Retry INFO logging on first failover from a proxy
[ https://issues.apache.org/jira/browse/HADOOP-17116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-17116: --- Issue Type: Bug (was: Task) > Skip Retry INFO logging on first failover from a proxy > -- > > Key: HADOOP-17116 > URL: https://issues.apache.org/jira/browse/HADOOP-17116 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Blocker > Attachments: HADOOP-17116.001.patch > > > RetryInvocationHandler logs an INFO level message on every failover except > the first. This used to be ideal before when there were only 2 proxies in the > FailoverProxyProvider. But if there are more than 2 proxies (as is possible > with 3 or more NNs in HA), then there could be more than one failover to find > the currently active proxy. > To avoid creating noise in clients logs/ console, RetryInvocationHandler > should skip logging once for each proxy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17116) Skip Retry INFO logging on first failover from a proxy
[ https://issues.apache.org/jira/browse/HADOOP-17116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-17116: --- Status: Patch Available (was: Open) > Skip Retry INFO logging on first failover from a proxy > -- > > Key: HADOOP-17116 > URL: https://issues.apache.org/jira/browse/HADOOP-17116 > Project: Hadoop Common > Issue Type: Task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HADOOP-17116.001.patch > > > RetryInvocationHandler logs an INFO level message on every failover except > the first. This used to be ideal before when there were only 2 proxies in the > FailoverProxyProvider. But if there are more than 2 proxies (as is possible > with 3 or more NNs in HA), then there could be more than one failover to find > the currently active proxy. > To avoid creating noise in clients logs/ console, RetryInvocationHandler > should skip logging once for each proxy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15524) BytesWritable causes OOME when array size reaches Integer.MAX_VALUE
[
https://issues.apache.org/jira/browse/HADOOP-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104513#comment-17104513
]
Arpit Agarwal commented on HADOOP-15524:
Hey [~nanda] can you take a quick look and confirm you are still +1 on this
change? I probably missed the notification when you tagged me last time.
> BytesWritable causes OOME when array size reaches Integer.MAX_VALUE
> ---
>
> Key: HADOOP-15524
> URL: https://issues.apache.org/jira/browse/HADOOP-15524
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Reporter: Joseph Smith
>Assignee: Joseph Smith
>Priority: Major
>
> BytesWritable.setSize uses Integer.MAX_VALUE to initialize the internal
> array. On my environment, this causes an OOME
> {code:java}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> {code}
> byte[Integer.MAX_VALUE-2] must be used to prevent this error.
> Tested on OSX and CentOS 7 using Java version 1.8.0_131.
> I noticed that java.util.ArrayList contains the following
> {code:java}
> /**
> * The maximum size of array to allocate.
> * Some VMs reserve some header words in an array.
> * Attempts to allocate larger arrays may result in
> * OutOfMemoryError: Requested array size exceeds VM limit
> */
> private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
> {code}
>
> BytesWritable.setSize should use something similar to prevent an OOME from
> occurring.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15524) BytesWritable causes OOME when array size reaches Integer.MAX_VALUE
[
https://issues.apache.org/jira/browse/HADOOP-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal reassigned HADOOP-15524:
--
Assignee: Joseph Smith
> BytesWritable causes OOME when array size reaches Integer.MAX_VALUE
> ---
>
> Key: HADOOP-15524
> URL: https://issues.apache.org/jira/browse/HADOOP-15524
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Reporter: Joseph Smith
>Assignee: Joseph Smith
>Priority: Major
>
> BytesWritable.setSize uses Integer.MAX_VALUE to initialize the internal
> array. On my environment, this causes an OOME
> {code:java}
> Exception in thread "main" java.lang.OutOfMemoryError: Requested array size
> exceeds VM limit
> {code}
> byte[Integer.MAX_VALUE-2] must be used to prevent this error.
> Tested on OSX and CentOS 7 using Java version 1.8.0_131.
> I noticed that java.util.ArrayList contains the following
> {code:java}
> /**
> * The maximum size of array to allocate.
> * Some VMs reserve some header words in an array.
> * Attempts to allocate larger arrays may result in
> * OutOfMemoryError: Requested array size exceeds VM limit
> */
> private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
> {code}
>
> BytesWritable.setSize should use something similar to prevent an OOME from
> occurring.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16992) Update download links
Arpit Agarwal created HADOOP-16992: -- Summary: Update download links Key: HADOOP-16992 URL: https://issues.apache.org/jira/browse/HADOOP-16992 Project: Hadoop Common Issue Type: Improvement Components: website Reporter: Arpit Agarwal The download lists for signatures/checksums/KEYS should be updated from dist.apache.org to https://downloads.apache.org/hadoop/ozone/. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-16833:
---
Fix Version/s: 3.3.0
Target Version/s: (was: 3.3.0)
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the contribution [~sodonnell]. I've committed this to trunk.
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL: https://issues.apache.org/jira/browse/HADOOP-16833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16833.001.patch, HADOOP-16833.002.patch,
> HADOOP-16833.003.patch, HADOOP-16833.004.patch, HADOOP-16833.005.patch,
> HADOOP-16833.006.patch
>
>
> Within the Datanode, we often see contention around the FsDatasetImpl lock.
> This can be for various reasons, eg the DN is under IO load, and other bugs,
> eg HDFS-15131.
> When DN slow downs happen, it is very difficult to debug what is causing it,
> as there are few messages in the logs which indicate what is happening.
> In my experience, the presence of this log is informative:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> This comes from the datanode heartbeat thread, and it is logged when the
> commands returned from the heartbeat cannot be enqueued, and usually this is
> because the thread cannot get the FsDatasetImpl lock due to contention with
> other slow threads.
> HDFS-14997 moved the command processing to an async thread, and hence this
> useful message will disappear.
> InstrumentedLock introduced a feature to the datanode, where it will log a
> stack trace if any thread holds the lock for over 300ms. However this will
> not catch a scenario where 10 threads each hold a lock for 200ms, leading to
> the next in the queue having waited over 2 seconds.
> I believe it would be useful to extend InstrumentedLock to log if a thread
> has to wait for over some threshold. That way, we could be able to catch
> scenarios like the heartbeat thread shows us, but in a much more obvious way,
> provided lock contention is involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039281#comment-17039281
]
Arpit Agarwal commented on HADOOP-16833:
+1 for the v6 patch. I will commit it shortly.
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL: https://issues.apache.org/jira/browse/HADOOP-16833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HADOOP-16833.001.patch, HADOOP-16833.002.patch,
> HADOOP-16833.003.patch, HADOOP-16833.004.patch, HADOOP-16833.005.patch,
> HADOOP-16833.006.patch
>
>
> Within the Datanode, we often see contention around the FsDatasetImpl lock.
> This can be for various reasons, eg the DN is under IO load, and other bugs,
> eg HDFS-15131.
> When DN slow downs happen, it is very difficult to debug what is causing it,
> as there are few messages in the logs which indicate what is happening.
> In my experience, the presence of this log is informative:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> This comes from the datanode heartbeat thread, and it is logged when the
> commands returned from the heartbeat cannot be enqueued, and usually this is
> because the thread cannot get the FsDatasetImpl lock due to contention with
> other slow threads.
> HDFS-14997 moved the command processing to an async thread, and hence this
> useful message will disappear.
> InstrumentedLock introduced a feature to the datanode, where it will log a
> stack trace if any thread holds the lock for over 300ms. However this will
> not catch a scenario where 10 threads each hold a lock for 200ms, leading to
> the next in the queue having waited over 2 seconds.
> I believe it would be useful to extend InstrumentedLock to log if a thread
> has to wait for over some threshold. That way, we could be able to catch
> scenarios like the heartbeat thread shows us, but in a much more obvious way,
> provided lock contention is involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032041#comment-17032041
]
Arpit Agarwal commented on HADOOP-16833:
The v6 patch looks pretty good to me. I think the added synchronization is okay
as it will only be paid when the message is logged, which is once per
threshold. So the cost will be minimal.
I want to go over it once more before I +1 it.
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL: https://issues.apache.org/jira/browse/HADOOP-16833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HADOOP-16833.001.patch, HADOOP-16833.002.patch,
> HADOOP-16833.003.patch, HADOOP-16833.004.patch, HADOOP-16833.005.patch,
> HADOOP-16833.006.patch
>
>
> Within the Datanode, we often see contention around the FsDatasetImpl lock.
> This can be for various reasons, eg the DN is under IO load, and other bugs,
> eg HDFS-15131.
> When DN slow downs happen, it is very difficult to debug what is causing it,
> as there are few messages in the logs which indicate what is happening.
> In my experience, the presence of this log is informative:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> This comes from the datanode heartbeat thread, and it is logged when the
> commands returned from the heartbeat cannot be enqueued, and usually this is
> because the thread cannot get the FsDatasetImpl lock due to contention with
> other slow threads.
> HDFS-14997 moved the command processing to an async thread, and hence this
> useful message will disappear.
> InstrumentedLock introduced a feature to the datanode, where it will log a
> stack trace if any thread holds the lock for over 300ms. However this will
> not catch a scenario where 10 threads each hold a lock for 200ms, leading to
> the next in the queue having waited over 2 seconds.
> I believe it would be useful to extend InstrumentedLock to log if a thread
> has to wait for over some threshold. That way, we could be able to catch
> scenarios like the heartbeat thread shows us, but in a much more obvious way,
> provided lock contention is involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16833) InstrumentedLock should log lock queue time
[
https://issues.apache.org/jira/browse/HADOOP-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030201#comment-17030201
]
Arpit Agarwal commented on HADOOP-16833:
Thanks for this improvement and the offline explanation [~sodonnell] .
The {{check}} method changes are a bit tricky. Lockless synchronization is a
minefield! I realized that the original implementation has a bug which is
carried over with these changes.The {{getAndSet}} after the while loop ends can
lose changes made by another thread.
{code:java}
} while (!lastLogTime.compareAndSet(localLastLogTs, now)); << Thread 1
successfully does compareAndSet
<< Thread 2 sees it cannot log yet, so it updates
suppressed and maxWait.
long suppressed = suppressedCounter.getAndSet(0); << Thread 1 resets
changes made by thread 1.
long maxWait = maxWaitCounter.getAndSet(0);
{code}
I don't see an easy solution. we are attempting to update multiple variables
atomically without getting a lock that covers all the updates. We can make a
decision that the logging is not making a guarantee of absolute precision and
we can potentially lose a few samples.
> InstrumentedLock should log lock queue time
> ---
>
> Key: HADOOP-16833
> URL: https://issues.apache.org/jira/browse/HADOOP-16833
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Affects Versions: 3.3.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HADOOP-16833.001.patch, HADOOP-16833.002.patch,
> HADOOP-16833.003.patch, HADOOP-16833.004.patch
>
>
> Within the Datanode, we often see contention around the FsDatasetImpl lock.
> This can be for various reasons, eg the DN is under IO load, and other bugs,
> eg HDFS-15131.
> When DN slow downs happen, it is very difficult to debug what is causing it,
> as there are few messages in the logs which indicate what is happening.
> In my experience, the presence of this log is informative:
> {code}
> 2019-11-27 09:00:49,678 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 21540ms to process 1 commands from NN
> {code}
> This comes from the datanode heartbeat thread, and it is logged when the
> commands returned from the heartbeat cannot be enqueued, and usually this is
> because the thread cannot get the FsDatasetImpl lock due to contention with
> other slow threads.
> HDFS-14997 moved the command processing to an async thread, and hence this
> useful message will disappear.
> InstrumentedLock introduced a feature to the datanode, where it will log a
> stack trace if any thread holds the lock for over 300ms. However this will
> not catch a scenario where 10 threads each hold a lock for 200ms, leading to
> the next in the queue having waited over 2 seconds.
> I believe it would be useful to extend InstrumentedLock to log if a thread
> has to wait for over some threshold. That way, we could be able to catch
> scenarios like the heartbeat thread shows us, but in a much more obvious way,
> provided lock contention is involved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16688) Update Hadoop website to mention Ozone mailing lists
[ https://issues.apache.org/jira/browse/HADOOP-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-16688: --- Resolution: Fixed Status: Resolved (was: Patch Available) +1 Merged via GitHub. Thanks [~ayushtkn]! > Update Hadoop website to mention Ozone mailing lists > > > Key: HADOOP-16688 > URL: https://issues.apache.org/jira/browse/HADOOP-16688 > Project: Hadoop Common > Issue Type: Improvement > Components: website >Reporter: Arpit Agarwal >Priority: Major > > Now that Ozone has its separate mailing lists, let's list them on the Hadoop > website. > https://hadoop.apache.org/mailing_lists.html > Thanks to [~ayushtkn] for suggesting this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16688) Update Hadoop website to mention Ozone mailing lists
Arpit Agarwal created HADOOP-16688: -- Summary: Update Hadoop website to mention Ozone mailing lists Key: HADOOP-16688 URL: https://issues.apache.org/jira/browse/HADOOP-16688 Project: Hadoop Common Issue Type: Improvement Components: website Reporter: Arpit Agarwal Now that Ozone has its separate mailing lists, let's list them on the Hadoop website. https://hadoop.apache.org/mailing_lists.html Thanks to [~ayushtkn] for suggesting this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-16654) Delete hadoop-ozone and hadoop-hdds subprojects from apache trunk
[ https://issues.apache.org/jira/browse/HADOOP-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal moved HDDS-2288 to HADOOP-16654: -- Key: HADOOP-16654 (was: HDDS-2288) Target Version/s: 3.3.0 (was: 0.5.0) Workflow: no-reopen-closed, patch-avail (was: patch-available, re-open possible) Project: Hadoop Common (was: Hadoop Distributed Data Store) > Delete hadoop-ozone and hadoop-hdds subprojects from apache trunk > - > > Key: HADOOP-16654 > URL: https://issues.apache.org/jira/browse/HADOOP-16654 > Project: Hadoop Common > Issue Type: Task >Reporter: Marton Elek >Priority: Major > > As described in the HDDS-2287 ozone/hdds sources are moving to the > apache/hadoop-ozone git repository. > All the remaining ozone/hdds files can be removed from trunk (including hdds > profile in main pom.xml) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x
[ https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-16152: --- Description: Some big data projects have been upgraded Jetty to 9.4.x, which causes some compatibility issues. Spark: [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146] Calcite: [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87] Hive: HIVE-21211 was: Some big data projects have been upgraded Jetty to 9.4.x, which causes some compatibility issues. Spark: [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146] Calcite: [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87] Hive: https://issues.apache.org/jira/browse/HIVE-21211 > Upgrade Eclipse Jetty version to 9.4.x > -- > > Key: HADOOP-16152 > URL: https://issues.apache.org/jira/browse/HADOOP-16152 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Yuming Wang >Assignee: Siyao Meng >Priority: Major > Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, > HADOOP-16152.v1.patch > > > Some big data projects have been upgraded Jetty to 9.4.x, which causes some > compatibility issues. > Spark: > [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146] > Calcite: > [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87] > Hive: HIVE-21211 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16350) Ability to tell HDFS client not to request KMS Information from NameNode
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869965#comment-16869965
]
Arpit Agarwal commented on HADOOP-16350:
bq. Arpit Agarwal do you mean cache it to UGI credentials?
No I meant the result of looking up the boolean from config. I think it will be
ok to look that up once per connection, however I would like it a bit better if
the value was looked up once and cached subsequently.
bq. However, with a single boolean, how to we differentiate local vs remote NN
and their KMS configuration? With the new Hadoop clients where the
recommendation is to be configureless wrt. local kms and letting NN tell the
right KMS uri. This may be an issue as we will ignore all KMS returned from
NNs, no matter is is local or remote.
[~xyao], thanks for catching that. You are right that there is no good way to
distinguish local vs remote NN. This setting will affect all connections. Does
the documentation update in the v5 patch address this concern, or should we add
a stronger warning?
> Ability to tell HDFS client not to request KMS Information from NameNode
>
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.00.patch, HADOOP-16350.01.patch,
> HADOOP-16350.02.patch, HADOOP-16350.03.patch, HADOOP-16350.04.patch,
> HADOOP-16350.05.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt,
[jira] [Updated] (HADOOP-16350) Ability to tell HDFS client not to request KMS Information from NameNode
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-16350:
---
Summary: Ability to tell HDFS client not to request KMS Information from
NameNode (was: Ability to tell Hadoop not to request KMS Information from
NameNode)
> Ability to tell HDFS client not to request KMS Information from NameNode
>
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.00.patch, HADOOP-16350.01.patch,
> HADOOP-16350.02.patch, HADOOP-16350.03.patch, HADOOP-16350.04.patch,
> HADOOP-16350.05.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 556079 for gss2002 on
[jira] [Updated] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from NameNode
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-16350:
---
Summary: Ability to tell Hadoop not to request KMS Information from
NameNode (was: Ability to tell Hadoop not to request KMS Information from
Remote NN )
> Ability to tell Hadoop not to request KMS Information from NameNode
> ---
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.00.patch, HADOOP-16350.01.patch,
> HADOOP-16350.02.patch, HADOOP-16350.03.patch, HADOOP-16350.04.patch,
> HADOOP-16350.05.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 556079 for gss2002 on ha-hdfs:tech
> 19/05/29
[jira] [Commented] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16869669#comment-16869669
]
Arpit Agarwal commented on HADOOP-16350:
The v5 patch looks good. I still have one question: should we cache the result
of the config lookup?
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.00.patch, HADOOP-16350.01.patch,
> HADOOP-16350.02.patch, HADOOP-16350.03.patch, HADOOP-16350.04.patch,
> HADOOP-16350.05.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 556079 for gss2002 on ha-hdfs:tech
> 19/05/29 14:06:10
[jira] [Commented] (HADOOP-16350) Ability to tell Hadoop not to request KMS Information from Remote NN
[
https://issues.apache.org/jira/browse/HADOOP-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868752#comment-16868752
]
Arpit Agarwal commented on HADOOP-16350:
The v02 patch looks pretty good to me. Would it be a good idea to cache the
result of the config lookup? I am not sure if there could be a perf impact.
There is a checkstyle issue that needs to be addressed.
> Ability to tell Hadoop not to request KMS Information from Remote NN
> -
>
> Key: HADOOP-16350
> URL: https://issues.apache.org/jira/browse/HADOOP-16350
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 2.8.3, 3.0.0, 2.7.6, 3.1.2
>Reporter: Greg Senia
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16350.00.patch, HADOOP-16350.01.patch,
> HADOOP-16350.02.patch
>
>
> Before HADOOP-14104 Remote KMSServer URIs were not requested from the remote
> NameNode and their associated remote KMSServer delegation token. Many
> customers were using this as a security feature to prevent TDE/Encryption
> Zone data from being distcped to remote clusters. But there was still a use
> case to allow distcp of data residing in folders that are not being encrypted
> with a KMSProvider/Encrypted Zone.
> So after upgrading to a version of Hadoop that contained HADOOP-14104 distcp
> now fails as we along with other customers (HDFS-13696) DO NOT allow
> KMSServer endpoints to be exposed out of our cluster network as data residing
> in these TDE/Zones contain very critical data that cannot be distcped between
> clusters.
> I propose adding a new code block with the following custom property
> "hadoop.security.kms.client.allow.remote.kms" it will default to "true" so
> keeping current feature of HADOOP-14104 but if specified to "false" will
> allow this area of code to operate as it did before HADOOP-14104. I can see
> the value in HADOOP-14104 but the way Hadoop worked before this JIRA/Issue
> should of at least had an option specified to allow Hadoop/KMS code to
> operate similar to how it did before by not requesting remote KMSServer URIs
> which would than attempt to get a delegation token even if not operating on
> encrypted zones.
> Error when KMS Server traffic is not allowed between cluster networks per
> enterprise security standard which cannot be changed they denied the request
> for exception so the only solution is to allow a feature to not attempt to
> request tokens.
> {code:java}
> $ hadoop distcp -Ddfs.namenode.kerberos.principal.pattern=*
> -Dmapreduce.job.hdfs-servers.token-renewal.exclude=tech
> hdfs:///processed/public/opendata/samples/distcp_test/distcp_file.txt
> hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt
> 19/05/29 14:06:09 INFO tools.DistCp: Input Options: DistCpOptions
> {atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, overwrite=false, append=false, useDiff=false,
> fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true,
> numListstatusThreads=0, maxMaps=20, mapBandwidth=100,
> sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[],
> preserveRawXattrs=false, atomicWorkPath=null, logPath=null,
> sourceFileListing=null,
> sourcePaths=[hdfs:/processed/public/opendata/samples/distcp_test/distcp_file.txt],
>
> targetPath=hdfs://tech/processed/public/opendata/samples/distcp_test/distcp_file2.txt,
> targetPathExists=true, filtersFile='null', verboseLog=false}
> 19/05/29 14:06:09 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 5093920 for gss2002 on ha-hdfs:unit
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> HDFS_DELEGATION_TOKEN, Service: ha-hdfs:unit, Ident: (HDFS_DELEGATION_TOKEN
> token 5093920 for gss2002)
> 19/05/29 14:06:10 INFO security.TokenCache: Got dt for hdfs://unit; Kind:
> kms-dt, Service: ha21d53en.unit.hdp.example.com:9292, Ident: (owner=gss2002,
> renewer=yarn, realUser=, issueDate=1559153170120, maxDate=1559757970120,
> sequenceNumber=237, masterKeyId=2)
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 1;
> dirCnt = 0
> 19/05/29 14:06:10 INFO tools.SimpleCopyListing: Build file listing completed.
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO tools.DistCp: Number of paths in the copy list: 1
> 19/05/29 14:06:10 INFO client.AHSProxy: Connecting to Application History
> server at ha21d53mn.unit.hdp.example.com/10.70.49.2:10200
> 19/05/29 14:06:10 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 556079 for gss2002 on
[jira] [Commented] (HADOOP-16325) Add ability to run python test and build docker in docker in start-build-env.sh
[
https://issues.apache.org/jira/browse/HADOOP-16325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847234#comment-16847234
]
Arpit Agarwal commented on HADOOP-16325:
Hi [~eyang], I assume this should be tested with
{code}
mvn clean install package -Pdist,docker
{code}
> Add ability to run python test and build docker in docker in
> start-build-env.sh
> ---
>
> Key: HADOOP-16325
> URL: https://issues.apache.org/jira/browse/HADOOP-16325
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HADOOP-16325.001.patch, HADOOP-16325.002.patch,
> HADOOP-16325.003.patch
>
>
> Ozone uses docker-compose, pytest and blockade to simulate network failure.
> It would be great to have ability to run these integration test tools in the
> developer docker environment.
> Ozone and YARN have optional profiles to build docker images using -Pdocker.
> It would be a good addition to have ability to build docker image inside the
> developer docker environment as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13386) Upgrade Avro to 1.8.x
[ https://issues.apache.org/jira/browse/HADOOP-13386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825266#comment-16825266 ] Arpit Agarwal commented on HADOOP-13386: [~Jantner] I've added you as a contributor and assigned this to you. Thanks for creating the pull request. Could you please summarize the testing you've done to validate this change? Also Steve had a question about transitive dependencies. > Upgrade Avro to 1.8.x > - > > Key: HADOOP-13386 > URL: https://issues.apache.org/jira/browse/HADOOP-13386 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Ben McCann >Assignee: Kalman >Priority: Major > > Avro 1.8.x makes generated classes serializable which makes them much easier > to use with Spark. It would be great to upgrade Avro to 1.8.x -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13386) Upgrade Avro to 1.8.x
[ https://issues.apache.org/jira/browse/HADOOP-13386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HADOOP-13386: -- Assignee: Kalman > Upgrade Avro to 1.8.x > - > > Key: HADOOP-13386 > URL: https://issues.apache.org/jira/browse/HADOOP-13386 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Ben McCann >Assignee: Kalman >Priority: Major > > Avro 1.8.x makes generated classes serializable which makes them much easier > to use with Spark. It would be great to upgrade Avro to 1.8.x -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13386) Upgrade Avro to 1.8.x
[ https://issues.apache.org/jira/browse/HADOOP-13386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reopened HADOOP-13386: Reopening since HADOOP-14992 upgraded Avro to 1.7.7. This jira requested an upgrade to 1.8.x. > Upgrade Avro to 1.8.x > - > > Key: HADOOP-13386 > URL: https://issues.apache.org/jira/browse/HADOOP-13386 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Ben McCann >Priority: Major > > Avro 1.8.x makes generated classes serializable which makes them much easier > to use with Spark. It would be great to upgrade Avro to 1.8.x -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16126) ipc.Client.stop() may sleep too long to wait for all connections
[
https://issues.apache.org/jira/browse/HADOOP-16126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777292#comment-16777292
]
Arpit Agarwal commented on HADOOP-16126:
+1
> ipc.Client.stop() may sleep too long to wait for all connections
>
>
> Key: HADOOP-16126
> URL: https://issues.apache.org/jira/browse/HADOOP-16126
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Major
> Attachments: c16126_20190219.patch, c16126_20190220.patch,
> c16126_20190221.patch
>
>
> {code}
> //Client.java
> public void stop() {
> ...
> // wait until all connections are closed
> while (!connections.isEmpty()) {
> try {
> Thread.sleep(100);
> } catch (InterruptedException e) {
> }
> }
> ...
> }
> {code}
> In the code above, the sleep time is 100ms. We found that simply changing
> the sleep time to 10ms could improve a Hive job running time by 10x.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11722) Some Instances of Services using ZKDelegationTokenSecretManager go down when old token cannot be deleted
[
https://issues.apache.org/jira/browse/HADOOP-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758686#comment-16758686
]
Arpit Agarwal commented on HADOOP-11722:
It looks like this was never actually committed to branch-2.7.
> Some Instances of Services using ZKDelegationTokenSecretManager go down when
> old token cannot be deleted
>
>
> Key: HADOOP-11722
> URL: https://issues.apache.org/jira/browse/HADOOP-11722
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Fix For: 2.7.0
>
> Attachments: HADOOP-11722.1.patch, HADOOP-11722.2.patch
>
>
> The delete node code in {{ZKDelegationTokenSecretManager}} is as follows :
> {noformat}
>while(zkClient.checkExists().forPath(nodeRemovePath) != null){
> zkClient.delete().guaranteed().forPath(nodeRemovePath);
>}
> {noformat}
> When instances of a Service using {{ZKDelegationTokenSecretManager}} try
> deleting a node simutaneously, It is possible that all of them enter into the
> while loop in which case, all peers will try to delete the node.. Only 1 will
> succeed and the rest will throw an exception.. which will bring down the node.
> The Exception is as follows :
> {noformat}
> 2015-03-15 10:24:54,000 ERROR
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> ExpiredTokenRemover thread received unexpected exception
> java.lang.RuntimeException: Could not remove Stored Token
> ZKDTSMDelegationToken_28
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.removeStoredToken(ZKDelegationTokenSecretManager.java:770)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.removeExpiredToken(AbstractDelegationTokenSecretManager.java:605)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.access$400(AbstractDelegationTokenSecretManager.java:54)
> at
> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$ExpiredTokenRemover.run(AbstractDelegationTokenSecretManager.java:656)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_28
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> at
> org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:238)
> at
> org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:233)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
> at
> org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:230)
> at
> org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:214)
> at
> org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41)
> at
> org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.removeStoredToken(ZKDelegationTokenSecretManager.java:764)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16071) Fix typo in DistCp Counters - Bandwidth in Bytes
[
https://issues.apache.org/jira/browse/HADOOP-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751502#comment-16751502
]
Arpit Agarwal commented on HADOOP-16071:
I think Steve is talking about any automation that parses the text output of
distcp.
> Fix typo in DistCp Counters - Bandwidth in Bytes
>
>
> Key: HADOOP-16071
> URL: https://issues.apache.org/jira/browse/HADOOP-16071
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.2.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16071.001.patch
>
>
> {code:bash|title=DistCp MR Job Counters}
> ...
> DistCp Counters
> Bandwidth in Btyes=20971520
> Bytes Copied=20971520
> Bytes Expected=20971520
> Files Copied=1
> {code}
> {noformat}
> Bandwidth in Btyes -> Bandwidth in Bytes
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16071) Fix typo in DistCp Counters - Bandwidth in Bytes
[
https://issues.apache.org/jira/browse/HADOOP-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751502#comment-16751502
]
Arpit Agarwal edited comment on HADOOP-16071 at 1/24/19 8:13 PM:
-
I think Steve is talking about any automation that parses the text output of
distcp. That may break if it is looking for the specific output string with the
typo.
was (Author: arpitagarwal):
I think Steve is talking about any automation that parses the text output of
distcp.
> Fix typo in DistCp Counters - Bandwidth in Bytes
>
>
> Key: HADOOP-16071
> URL: https://issues.apache.org/jira/browse/HADOOP-16071
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.2.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16071.001.patch
>
>
> {code:bash|title=DistCp MR Job Counters}
> ...
> DistCp Counters
> Bandwidth in Btyes=20971520
> Bytes Copied=20971520
> Bytes Expected=20971520
> Files Copied=1
> {code}
> {noformat}
> Bandwidth in Btyes -> Bandwidth in Bytes
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16071) Fix typo in DistCp Counters - Bandwidth in Bytes
[
https://issues.apache.org/jira/browse/HADOOP-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751328#comment-16751328
]
Arpit Agarwal commented on HADOOP-16071:
+1
> Fix typo in DistCp Counters - Bandwidth in Bytes
>
>
> Key: HADOOP-16071
> URL: https://issues.apache.org/jira/browse/HADOOP-16071
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools/distcp
>Affects Versions: 3.2.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16071.001.patch
>
>
> {code:bash|title=DistCp MR Job Counters}
> ...
> DistCp Counters
> Bandwidth in Btyes=20971520
> Bytes Copied=20971520
> Bytes Expected=20971520
> Files Copied=1
> {code}
> {noformat}
> Bandwidth in Btyes -> Bandwidth in Bytes
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12558) distcp documentation is woefully out of date
[ https://issues.apache.org/jira/browse/HADOOP-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-12558: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.1 3.1.2 3.3.0 Status: Resolved (was: Patch Available) I've committed this. Thanks [~dineshchitlangia]. > distcp documentation is woefully out of date > > > Key: HADOOP-12558 > URL: https://issues.apache.org/jira/browse/HADOOP-12558 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, tools/distcp >Reporter: Allen Wittenauer >Assignee: Dinesh Chitlangia >Priority: Critical > Labels: newbie > Fix For: 3.3.0, 3.1.2, 3.2.1 > > Attachments: HADOOP-12558.001.patch, HADOOP-12558.002.patch, > HADOOP-12558.003.patch > > > There are a ton of distcp tune-ables that have zero documentation outside of > the source code. This should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12558) distcp documentation is woefully out of date
[ https://issues.apache.org/jira/browse/HADOOP-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688656#comment-16688656 ] Arpit Agarwal commented on HADOOP-12558: +1 pending Jenkins. > distcp documentation is woefully out of date > > > Key: HADOOP-12558 > URL: https://issues.apache.org/jira/browse/HADOOP-12558 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, tools/distcp >Reporter: Allen Wittenauer >Assignee: Dinesh Chitlangia >Priority: Critical > Labels: newbie > Attachments: HADOOP-12558.001.patch, HADOOP-12558.002.patch, > HADOOP-12558.003.patch > > > There are a ton of distcp tune-ables that have zero documentation outside of > the source code. This should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15936) [JDK 11] MiniDFSClusterManager & MiniHadoopClusterManager compilation fails due to the usage of '_' as identifier
[
https://issues.apache.org/jira/browse/HADOOP-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15936:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
+1 I've committed this. The Jenkins failures are clearly unrelated.
Thanks [~zvenczel].
> [JDK 11] MiniDFSClusterManager & MiniHadoopClusterManager compilation fails
> due to the usage of '_' as identifier
> -
>
> Key: HADOOP-15936
> URL: https://issues.apache.org/jira/browse/HADOOP-15936
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
> Environment: openjdk version "11" 2018-09-25
>Reporter: Devaraj K
>Assignee: Zsolt Venczel
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15936.01.patch
>
>
> {code:xml}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java:[130,37]
> as of release 9, '_' is a keyword, and may not be used as an identifier
> [INFO] 1 error
> {code}
> {code:xml}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java:[140,37]
> as of release 9, '_' is a keyword, and may not be used as an identifier
> [INFO] 1 error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12558) distcp documentation is woefully out of date
[
https://issues.apache.org/jira/browse/HADOOP-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688613#comment-16688613
]
Arpit Agarwal commented on HADOOP-12558:
Thanks [~dineshchitlangia]. It looks like the xtrack option is tagged as
unstable. Perhaps we should mention it is experimental, and also that it cannot
be used with ATOMIC_COMMIT.
{code}
/**
* Track missing files in target that are missing from source
* This allows for other applications to complete the synchronization,
* possibly with object-store-specific delete algorithms.
* Typically used in conjunction with SYNC_FOLDERS
* Incompatible with ATOMIC_COMMIT
*/
@InterfaceStability.Unstable
TRACK_MISSING(DistCpConstants.CONF_LABEL_TRACK_MISSING,
new Option("xtrack", true,
"Save information about missing source files to the"
+ " specified directory")),
{code}
> distcp documentation is woefully out of date
>
>
> Key: HADOOP-12558
> URL: https://issues.apache.org/jira/browse/HADOOP-12558
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, tools/distcp
>Reporter: Allen Wittenauer
>Assignee: Dinesh Chitlangia
>Priority: Critical
> Labels: newbie
> Attachments: HADOOP-12558.001.patch, HADOOP-12558.002.patch
>
>
> There are a ton of distcp tune-ables that have zero documentation outside of
> the source code. This should be fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12558) distcp documentation is woefully out of date
[ https://issues.apache.org/jira/browse/HADOOP-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688432#comment-16688432 ] Arpit Agarwal commented on HADOOP-12558: Just looked at the patch and realized you are updating the generated site docs for past releases. Let's just update the in-build docs instead, so the option is documented for future releases. > distcp documentation is woefully out of date > > > Key: HADOOP-12558 > URL: https://issues.apache.org/jira/browse/HADOOP-12558 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, tools/distcp >Reporter: Allen Wittenauer >Assignee: Dinesh Chitlangia >Priority: Critical > Labels: newbie > Attachments: HADOOP-12558.001.patch > > > There are a ton of distcp tune-ables that have zero documentation outside of > the source code. This should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12558) distcp documentation is woefully out of date
[ https://issues.apache.org/jira/browse/HADOOP-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688426#comment-16688426 ] Arpit Agarwal commented on HADOOP-12558: Thanks [~dineshchitlangia]. Can you please rebase the patch to trunk? I will take a look. > distcp documentation is woefully out of date > > > Key: HADOOP-12558 > URL: https://issues.apache.org/jira/browse/HADOOP-12558 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, tools/distcp >Reporter: Allen Wittenauer >Assignee: Dinesh Chitlangia >Priority: Critical > Labels: newbie > Attachments: HADOOP-12558.001.patch > > > There are a ton of distcp tune-ables that have zero documentation outside of > the source code. This should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15901) IPC Client and Server should use Time.monotonicNow() for elapsed times.
[
https://issues.apache.org/jira/browse/HADOOP-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675702#comment-16675702
]
Arpit Agarwal commented on HADOOP-15901:
That is depressing to learn.
> IPC Client and Server should use Time.monotonicNow() for elapsed times.
> ---
>
> Key: HADOOP-15901
> URL: https://issues.apache.org/jira/browse/HADOOP-15901
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc, metrics
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Attachments: HADOOP-15901-01.patch
>
>
> Client.java and Server.java uses {{Time.now()}} to calculate the elapsed
> times/timeouts. This could result in undesired results when system clock's
> time changes.
> {{Time.monotonicNow()}} should be used for elapsed time calculations within
> same JVM.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15880) WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder
[ https://issues.apache.org/jira/browse/HADOOP-15880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663079#comment-16663079 ] Arpit Agarwal edited comment on HADOOP-15880 at 10/25/18 1:04 AM: -- Hi [~Sunilkc], I don't expect any of the connectors (E.g. WASB, S3A) to honor _fs.trash.interval_. The purge functionality works for HDFS because it is implemented in the HDFS namenode. The cloud object stores have no idea about this setting. was (Author: arpitagarwal): Hi [~Sunilkc], I don't expect any of the connectors (E.g. WASB, S3A) to honor _fs.trash.interval_. The purge functionality is implemented in the HDFS namenode. The cloud object stores have no idea about this setting. > WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder > -- > > Key: HADOOP-15880 > URL: https://issues.apache.org/jira/browse/HADOOP-15880 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.3 > Environment: Any HDInsigth cluster pointing to WASB. >Reporter: Sunil Kumar Chakrapani >Priority: Major > Labels: WASB > > when "fs.trash.interval" is set to a value, trash for the local hdfs got > cleared where as the trash folder on WASB doesn't get deleted and the files > get piled up on WASB store.. > WASB doesn't pick up fs.trash.interval value and this fails to auto purge > trash folder on WASB store. > > *Issue : WASB doesn't honor fs.trash.interval and this fails to auto purge > trash folder* > *Steps to reproduce Scenario:* > *Delete any file stored on HDFS* > hdfs dfs -D "fs.default.name=hdfs://mycluster/" -rm /hivestore.txt > 18/10/23 06:18:05 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://mycluster/hivestore.txt' to trash at: > hdfs://mycluster/user/sshuser/.Trash/Current/hivestore.txt > *When deleted the file is moved to trash folder* > hdfs dfs -rm wasb:///hivestore.txt > 18/10/23 06:19:13 INFO fs.TrashPolicyDefault: Moved: > 'wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/hivestore.txt' > to trash at: > wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt > *Reduced the fs.trash.interval from 360 to 1 and restarted all related > services.* > *Trash for the local hdfs gets cleared honoring the "fs.trash.interval" > value.* > hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls > hdfs://mycluster/user/sshuser/.Trash/Current/ > ls: File hdfs://mycluster/user/sshuser/.Trash/Current does not exist. > *Where as the trash for WASB doesn't get cleared.* > hdfs dfs -ls > wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/ > Found 1 items > -rw-r--r-- 1 sshuser supergroup 1084 2018-10-23 06:19 > wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15880) WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder
[ https://issues.apache.org/jira/browse/HADOOP-15880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663079#comment-16663079 ] Arpit Agarwal commented on HADOOP-15880: Hi [~Sunilkc], I don't expect any of the connectors (E.g. WASB, S3A) to honor _fs.trash.interval_. The purge functionality is implemented in the HDFS namenode. The cloud object stores have no idea about this setting. > WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder > -- > > Key: HADOOP-15880 > URL: https://issues.apache.org/jira/browse/HADOOP-15880 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.7.3 > Environment: Any HDInsigth cluster pointing to WASB. >Reporter: Sunil Kumar Chakrapani >Priority: Major > Labels: WASB > > when "fs.trash.interval" is set to a value, trash for the local hdfs got > cleared where as the trash folder on WASB doesn't get deleted and the files > get piled up on WASB store.. > WASB doesn't pick up fs.trash.interval value and this fails to auto purge > trash folder on WASB store. > > *Issue : WASB doesn't honor fs.trash.interval and this fails to auto purge > trash folder* > *Steps to reproduce Scenario:* > *Delete any file stored on HDFS* > hdfs dfs -D "fs.default.name=hdfs://mycluster/" -rm /hivestore.txt > 18/10/23 06:18:05 INFO fs.TrashPolicyDefault: Moved: > 'hdfs://mycluster/hivestore.txt' to trash at: > hdfs://mycluster/user/sshuser/.Trash/Current/hivestore.txt > *When deleted the file is moved to trash folder* > hdfs dfs -rm wasb:///hivestore.txt > 18/10/23 06:19:13 INFO fs.TrashPolicyDefault: Moved: > 'wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/hivestore.txt' > to trash at: > wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt > *Reduced the fs.trash.interval from 360 to 1 and restarted all related > services.* > *Trash for the local hdfs gets cleared honoring the "fs.trash.interval" > value.* > hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls > hdfs://mycluster/user/sshuser/.Trash/Current/ > ls: File hdfs://mycluster/user/sshuser/.Trash/Current does not exist. > *Where as the trash for WASB doesn't get cleared.* > hdfs dfs -ls > wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/ > Found 1 items > -rw-r--r-- 1 sshuser supergroup 1084 2018-10-23 06:19 > wasb://kcspark-2018-10-18t17-07-40-5...@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15867) Allow registering MBeans without additional jmx properties
[ https://issues.apache.org/jira/browse/HADOOP-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15867: --- Description: HDDS and Ozone use the MBeans.register overload added by HADOOP-15339. This is missing in Apache Hadoop 3.1.0 and earlier. This prevents us from building Ozone with earlier versions of Hadoop. More commonly, we see runtime exceptions if an earlier version of the Hadoop-common jar happens to be in the classpath. Let's add a reflection-based switch to invoke the right version of the API so we can build and use Ozone with Apache Hadoop 3.1.0. was: HDDS and Ozone use the MBeans.register overload added by HADOOP-15339. This is missing in Apache Hadoop 3.1.0 and earlier. This prevents us from building Ozone with earlier versions of Hadoop. More commonly, we see runtime exceptions if an earlier version of Hadoop happens to be in the classpath. Let's add a reflection-based switch to invoke the right version of the API so we can build and use Ozone with Apache Hadoop 3.1.0. > Allow registering MBeans without additional jmx properties > -- > > Key: HADOOP-15867 > URL: https://issues.apache.org/jira/browse/HADOOP-15867 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Blocker > > HDDS and Ozone use the MBeans.register overload added by HADOOP-15339. This > is missing in Apache Hadoop 3.1.0 and earlier. This prevents us from building > Ozone with earlier versions of Hadoop. More commonly, we see runtime > exceptions if an earlier version of the Hadoop-common jar happens to be in > the classpath. > Let's add a reflection-based switch to invoke the right version of the API so > we can build and use Ozone with Apache Hadoop 3.1.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15867) Allow registering MBeans without additional jmx properties
Arpit Agarwal created HADOOP-15867: -- Summary: Allow registering MBeans without additional jmx properties Key: HADOOP-15867 URL: https://issues.apache.org/jira/browse/HADOOP-15867 Project: Hadoop Common Issue Type: Improvement Components: metrics Reporter: Arpit Agarwal Assignee: Arpit Agarwal HDDS and Ozone use the MBeans.register overload added by HADOOP-15339. This is missing in Apache Hadoop 3.1.0 and earlier. This prevents us from building Ozone with earlier versions of Hadoop. More commonly, we see runtime exceptions if an earlier version of Hadoop happens to be in the classpath. Let's add a reflection-based switch to invoke the right version of the API so we can build and use Ozone with Apache Hadoop 3.1.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15849) Upgrade netty version
[ https://issues.apache.org/jira/browse/HADOOP-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15849: --- Status: Patch Available (was: Open) > Upgrade netty version > - > > Key: HADOOP-15849 > URL: https://issues.apache.org/jira/browse/HADOOP-15849 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15849.01.patch > > > We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 > release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15849) Upgrade netty version
[ https://issues.apache.org/jira/browse/HADOOP-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648230#comment-16648230 ] Arpit Agarwal commented on HADOOP-15849: +1 pending Jenkins. > Upgrade netty version > - > > Key: HADOOP-15849 > URL: https://issues.apache.org/jira/browse/HADOOP-15849 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15849.01.patch > > > We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 > release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15772) Remove the 'Path ... should be specified as a URI' warnings on startup
[
https://issues.apache.org/jira/browse/HADOOP-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15772:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.2
3.0.4
3.2.0
2.10.0
Target Version/s: (was: 3.3.0)
Status: Resolved (was: Patch Available)
I've committed this. Thanks for the contribution [~ayushtkn].
> Remove the 'Path ... should be specified as a URI' warnings on startup
> --
>
> Key: HADOOP-15772
> URL: https://issues.apache.org/jira/browse/HADOOP-15772
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Reporter: Arpit Agarwal
>Assignee: Ayush Saxena
>Priority: Major
> Labels: newbie
> Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HDFS-13921-01.patch, HDFS-13921-02.patch
>
>
> The following warnings are logged on service startup and are noise. It is
> perfectly valid to list local paths without using the URI syntax.
> {code:java}
> 2018-09-16 23:16:11,393 WARN common.Util (Util.java:stringAsURI(99)) - Path
> /hadoop/hdfs/namenode should be specified as a URI in configuration files.
> Please update hdfs configuration.{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15774) Discovery of HA servers
[ https://issues.apache.org/jira/browse/HADOOP-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621035#comment-16621035 ] Arpit Agarwal commented on HADOOP-15774: Hi [~elgoiri], this looks interesting. Do you have a proposal or design? > Discovery of HA servers > --- > > Key: HADOOP-15774 > URL: https://issues.apache.org/jira/browse/HADOOP-15774 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Íñigo Goiri >Priority: Major > > Currently, Hadoop relies on configuration files to specify the servers. > This requires maintaining these configuration files and propagating the > changes. > Hadoop should have a framework to provide discovery. > For example, in HDFS, we could define the Namenodes in a shared location and > the DNs would use the framework to find the Namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15772) Remove the 'Path ... should be specified as a URI' warnings on startup
[
https://issues.apache.org/jira/browse/HADOOP-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620722#comment-16620722
]
Arpit Agarwal commented on HADOOP-15772:
Moved to HADOOP since the code change is in common. Will retrigger a Jenkins
run for testing commons.
> Remove the 'Path ... should be specified as a URI' warnings on startup
> --
>
> Key: HADOOP-15772
> URL: https://issues.apache.org/jira/browse/HADOOP-15772
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Reporter: Arpit Agarwal
>Assignee: Ayush Saxena
>Priority: Major
> Labels: newbie
> Attachments: HDFS-13921-01.patch, HDFS-13921-02.patch
>
>
> The following warnings are logged on service startup and are noise. It is
> perfectly valid to list local paths without using the URI syntax.
> {code:java}
> 2018-09-16 23:16:11,393 WARN common.Util (Util.java:stringAsURI(99)) - Path
> /hadoop/hdfs/namenode should be specified as a URI in configuration files.
> Please update hdfs configuration.{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15772) Remove the 'Path ... should be specified as a URI' warnings on startup
[
https://issues.apache.org/jira/browse/HADOOP-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15772:
---
Description:
The following warnings are logged on service startup and are noise. It is
perfectly valid to list local paths without using the URI syntax.
{code:java}
2018-09-16 23:16:11,393 WARN common.Util (Util.java:stringAsURI(99)) - Path
/hadoop/hdfs/namenode should be specified as a URI in configuration files.
Please update hdfs configuration.{code}
was:
The following warnings are logged multiple times on NameNode startup and are
noise. It is perfectly valid to list local paths without using the URI syntax.
{code:java}
2018-09-16 23:16:11,393 WARN common.Util (Util.java:stringAsURI(99)) - Path
/hadoop/hdfs/namenode should be specified as a URI in configuration files.
Please update hdfs configuration.{code}
> Remove the 'Path ... should be specified as a URI' warnings on startup
> --
>
> Key: HADOOP-15772
> URL: https://issues.apache.org/jira/browse/HADOOP-15772
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Reporter: Arpit Agarwal
>Assignee: Ayush Saxena
>Priority: Major
> Labels: newbie
> Attachments: HDFS-13921-01.patch, HDFS-13921-02.patch
>
>
> The following warnings are logged on service startup and are noise. It is
> perfectly valid to list local paths without using the URI syntax.
> {code:java}
> 2018-09-16 23:16:11,393 WARN common.Util (Util.java:stringAsURI(99)) - Path
> /hadoop/hdfs/namenode should be specified as a URI in configuration files.
> Please update hdfs configuration.{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15772) Remove the 'Path ... should be specified as a URI' warnings on startup
[
https://issues.apache.org/jira/browse/HADOOP-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal moved HDFS-13921 to HADOOP-15772:
---
Target Version/s: 3.3.0 (was: 3.3.0)
Component/s: (was: namenode)
conf
Key: HADOOP-15772 (was: HDFS-13921)
Project: Hadoop Common (was: Hadoop HDFS)
> Remove the 'Path ... should be specified as a URI' warnings on startup
> --
>
> Key: HADOOP-15772
> URL: https://issues.apache.org/jira/browse/HADOOP-15772
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Reporter: Arpit Agarwal
>Assignee: Ayush Saxena
>Priority: Major
> Labels: newbie
> Attachments: HDFS-13921-01.patch, HDFS-13921-02.patch
>
>
> The following warnings are logged multiple times on NameNode startup and are
> noise. It is perfectly valid to list local paths without using the URI syntax.
> {code:java}
> 2018-09-16 23:16:11,393 WARN common.Util (Util.java:stringAsURI(99)) - Path
> /hadoop/hdfs/namenode should be specified as a URI in configuration files.
> Please update hdfs configuration.{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15727) Missing dependency errors from dist-tools-hooks-maker
[
https://issues.apache.org/jira/browse/HADOOP-15727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15727:
---
Issue Type: Bug (was: Improvement)
> Missing dependency errors from dist-tools-hooks-maker
> -
>
> Key: HADOOP-15727
> URL: https://issues.apache.org/jira/browse/HADOOP-15727
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.2.0
>Reporter: Arpit Agarwal
>Priority: Major
>
> Building Hadoop with -Pdist -Dtar generates the following errors. These don't
> stop the build from succeeding though.
> {code}
> ERROR: hadoop-azure has missing dependencies:
> jetty-util-ajax-9.3.19.v20170502.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> hadoop-yarn-common-3.2.0-SNAPSHOT.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> hadoop-hdfs-client-3.2.0-SNAPSHOT.jar
> ERROR: hadoop-resourceestimator has missing dependencies: okhttp-2.7.5.jar
> ERROR: hadoop-resourceestimator has missing dependencies: okio-1.6.0.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> jersey-client-1.19.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> guice-servlet-4.0.jar
> ERROR: hadoop-resourceestimator has missing dependencies: guice-4.0.jar
> ERROR: hadoop-resourceestimator has missing dependencies: aopalliance-1.0.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> jersey-guice-1.19.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> jackson-module-jaxb-annotations-2.9.5.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> jackson-jaxrs-json-provider-2.9.5.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> jackson-jaxrs-base-2.9.5.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> hadoop-yarn-api-3.2.0-SNAPSHOT.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> hadoop-yarn-server-resourcemanager-3.2.0-SNAPSHOT.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> jetty-util-ajax-9.3.19.v20170502.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> hadoop-yarn-server-common-3.2.0-SNAPSHOT.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> hadoop-yarn-registry-3.2.0-SNAPSHOT.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> commons-daemon-1.0.13.jar
> ERROR: hadoop-resourceestimator has missing dependencies: dnsjava-2.1.7.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> geronimo-jcache_1.0_spec-1.0-alpha-1.jar
> ERROR: hadoop-resourceestimator has missing dependencies: ehcache-3.3.1.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> HikariCP-java7-2.4.12.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> hadoop-yarn-server-applicationhistoryservice-3.2.0-SNAPSHOT.jar
> ERROR: hadoop-resourceestimator has missing dependencies: objenesis-1.0.jar
> ERROR: hadoop-resourceestimator has missing dependencies: fst-2.50.jar
> ERROR: hadoop-resourceestimator has missing dependencies: java-util-1.9.0.jar
> ERROR: hadoop-resourceestimator has missing dependencies: json-io-2.5.1.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> hadoop-yarn-server-web-proxy-3.2.0-SNAPSHOT.jar
> ERROR: hadoop-resourceestimator has missing dependencies:
> leveldbjni-all-1.8.jar
> ERROR: hadoop-resourceestimator has missing dependencies: javax.inject-1.jar
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15727) Missing dependency errors from dist-tools-hooks-maker
Arpit Agarwal created HADOOP-15727:
--
Summary: Missing dependency errors from dist-tools-hooks-maker
Key: HADOOP-15727
URL: https://issues.apache.org/jira/browse/HADOOP-15727
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 3.2.0
Reporter: Arpit Agarwal
Building Hadoop with -Pdist -Dtar generates the following errors. These don't
stop the build from succeeding though.
{code}
ERROR: hadoop-azure has missing dependencies:
jetty-util-ajax-9.3.19.v20170502.jar
ERROR: hadoop-resourceestimator has missing dependencies:
hadoop-yarn-common-3.2.0-SNAPSHOT.jar
ERROR: hadoop-resourceestimator has missing dependencies:
hadoop-hdfs-client-3.2.0-SNAPSHOT.jar
ERROR: hadoop-resourceestimator has missing dependencies: okhttp-2.7.5.jar
ERROR: hadoop-resourceestimator has missing dependencies: okio-1.6.0.jar
ERROR: hadoop-resourceestimator has missing dependencies: jersey-client-1.19.jar
ERROR: hadoop-resourceestimator has missing dependencies: guice-servlet-4.0.jar
ERROR: hadoop-resourceestimator has missing dependencies: guice-4.0.jar
ERROR: hadoop-resourceestimator has missing dependencies: aopalliance-1.0.jar
ERROR: hadoop-resourceestimator has missing dependencies: jersey-guice-1.19.jar
ERROR: hadoop-resourceestimator has missing dependencies:
jackson-module-jaxb-annotations-2.9.5.jar
ERROR: hadoop-resourceestimator has missing dependencies:
jackson-jaxrs-json-provider-2.9.5.jar
ERROR: hadoop-resourceestimator has missing dependencies:
jackson-jaxrs-base-2.9.5.jar
ERROR: hadoop-resourceestimator has missing dependencies:
hadoop-yarn-api-3.2.0-SNAPSHOT.jar
ERROR: hadoop-resourceestimator has missing dependencies:
hadoop-yarn-server-resourcemanager-3.2.0-SNAPSHOT.jar
ERROR: hadoop-resourceestimator has missing dependencies:
jetty-util-ajax-9.3.19.v20170502.jar
ERROR: hadoop-resourceestimator has missing dependencies:
hadoop-yarn-server-common-3.2.0-SNAPSHOT.jar
ERROR: hadoop-resourceestimator has missing dependencies:
hadoop-yarn-registry-3.2.0-SNAPSHOT.jar
ERROR: hadoop-resourceestimator has missing dependencies:
commons-daemon-1.0.13.jar
ERROR: hadoop-resourceestimator has missing dependencies: dnsjava-2.1.7.jar
ERROR: hadoop-resourceestimator has missing dependencies:
geronimo-jcache_1.0_spec-1.0-alpha-1.jar
ERROR: hadoop-resourceestimator has missing dependencies: ehcache-3.3.1.jar
ERROR: hadoop-resourceestimator has missing dependencies:
HikariCP-java7-2.4.12.jar
ERROR: hadoop-resourceestimator has missing dependencies:
hadoop-yarn-server-applicationhistoryservice-3.2.0-SNAPSHOT.jar
ERROR: hadoop-resourceestimator has missing dependencies: objenesis-1.0.jar
ERROR: hadoop-resourceestimator has missing dependencies: fst-2.50.jar
ERROR: hadoop-resourceestimator has missing dependencies: java-util-1.9.0.jar
ERROR: hadoop-resourceestimator has missing dependencies: json-io-2.5.1.jar
ERROR: hadoop-resourceestimator has missing dependencies:
hadoop-yarn-server-web-proxy-3.2.0-SNAPSHOT.jar
ERROR: hadoop-resourceestimator has missing dependencies: leveldbjni-all-1.8.jar
ERROR: hadoop-resourceestimator has missing dependencies: javax.inject-1.jar
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-12558) distcp documentation is woefully out of date
[ https://issues.apache.org/jira/browse/HADOOP-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reopened HADOOP-12558: Reopening to evaluate if this needs a fix and bumping priority. > distcp documentation is woefully out of date > > > Key: HADOOP-12558 > URL: https://issues.apache.org/jira/browse/HADOOP-12558 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, tools/distcp >Reporter: Allen Wittenauer >Priority: Major > Labels: newbie > > There are a ton of distcp tune-ables that have zero documentation outside of > the source code. This should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12558) distcp documentation is woefully out of date
[ https://issues.apache.org/jira/browse/HADOOP-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-12558: --- Priority: Critical (was: Major) > distcp documentation is woefully out of date > > > Key: HADOOP-12558 > URL: https://issues.apache.org/jira/browse/HADOOP-12558 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, tools/distcp >Reporter: Allen Wittenauer >Priority: Critical > Labels: newbie > > There are a ton of distcp tune-ables that have zero documentation outside of > the source code. This should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12558) distcp documentation is woefully out of date
[ https://issues.apache.org/jira/browse/HADOOP-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-12558: --- Labels: newbie (was: ) > distcp documentation is woefully out of date > > > Key: HADOOP-12558 > URL: https://issues.apache.org/jira/browse/HADOOP-12558 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, tools/distcp >Reporter: Allen Wittenauer >Priority: Major > Labels: newbie > > There are a ton of distcp tune-ables that have zero documentation outside of > the source code. This should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12558) distcp documentation is woefully out of date
[ https://issues.apache.org/jira/browse/HADOOP-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-12558: --- Affects Version/s: (was: 3.0.0-alpha1) > distcp documentation is woefully out of date > > > Key: HADOOP-12558 > URL: https://issues.apache.org/jira/browse/HADOOP-12558 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, tools/distcp >Reporter: Allen Wittenauer >Priority: Major > Labels: newbie > > There are a ton of distcp tune-ables that have zero documentation outside of > the source code. This should be fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15721) Disable checkstyle javadoc warnings for test classes
[
https://issues.apache.org/jira/browse/HADOOP-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15721:
---
Labels: newbie (was: )
> Disable checkstyle javadoc warnings for test classes
>
>
> Key: HADOOP-15721
> URL: https://issues.apache.org/jira/browse/HADOOP-15721
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Shane Kumpf
>Priority: Minor
> Labels: newbie
>
> The current checkstyle rules with throw a warning if the javadoc is missing
> for a test class which is of minimal value. We should consider disabling this
> check and allowing contributors to comment test classes as they see fit.
> Here is an example:
> {code:java}
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/MockLinuxContainerRuntime.java:27:public
> class MockLinuxContainerRuntime implements LinuxContainerRuntime {: Missing
> a Javadoc comment. [JavadocType]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15493) DiskChecker should handle disk full situation
[ https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596369#comment-16596369 ] Arpit Agarwal commented on HADOOP-15493: I've removed the target version. We have removed the disk space check for now so this is not as critical. > DiskChecker should handle disk full situation > - > > Key: HADOOP-15493 > URL: https://issues.apache.org/jira/browse/HADOOP-15493 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Critical > Attachments: HADOOP-15493.01.patch, HADOOP-15493.02.patch > > > DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is > writable. > However check should not fail when file creation fails due to disk being > full. This avoids marking full disks as _failed_. > Reported by [~kihwal] and [~daryn] in HADOOP-15450. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15493) DiskChecker should handle disk full situation
[ https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15493: --- Target Version/s: (was: 3.2.0) > DiskChecker should handle disk full situation > - > > Key: HADOOP-15493 > URL: https://issues.apache.org/jira/browse/HADOOP-15493 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Critical > Attachments: HADOOP-15493.01.patch, HADOOP-15493.02.patch > > > DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is > writable. > However check should not fail when file creation fails due to disk being > full. This avoids marking full disks as _failed_. > Reported by [~kihwal] and [~daryn] in HADOOP-15450. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15693) Typo in Secure Mode documentation
[ https://issues.apache.org/jira/browse/HADOOP-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593898#comment-16593898 ] Arpit Agarwal commented on HADOOP-15693: Thanks [~tlevine]. +1 pending Jenkins. I've added you as a Hadoop contributor and assigned to you. > Typo in Secure Mode documentation > - > > Key: HADOOP-15693 > URL: https://issues.apache.org/jira/browse/HADOOP-15693 > Project: Hadoop Common > Issue Type: Bug > Components: common > Environment: OpenBSD 6.3 GENERIC.MP#4 amd64 >Reporter: Thomas Levine >Assignee: Thomas Levine >Priority: Trivial > Labels: documentation > Attachments: HADOOP-15693.patch > > Original Estimate: 0h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15693) Typo in Secure Mode documentation
[ https://issues.apache.org/jira/browse/HADOOP-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HADOOP-15693: -- Assignee: Thomas Levine > Typo in Secure Mode documentation > - > > Key: HADOOP-15693 > URL: https://issues.apache.org/jira/browse/HADOOP-15693 > Project: Hadoop Common > Issue Type: Bug > Components: common > Environment: OpenBSD 6.3 GENERIC.MP#4 amd64 >Reporter: Thomas Levine >Assignee: Thomas Levine >Priority: Trivial > Labels: documentation > Attachments: HADOOP-15693.patch > > Original Estimate: 0h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15693) Typo in Secure Mode documentation
[ https://issues.apache.org/jira/browse/HADOOP-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15693: --- Status: Patch Available (was: Open) > Typo in Secure Mode documentation > - > > Key: HADOOP-15693 > URL: https://issues.apache.org/jira/browse/HADOOP-15693 > Project: Hadoop Common > Issue Type: Bug > Components: common > Environment: OpenBSD 6.3 GENERIC.MP#4 amd64 >Reporter: Thomas Levine >Assignee: Thomas Levine >Priority: Trivial > Labels: documentation > Attachments: HADOOP-15693.patch > > Original Estimate: 0h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14946) S3Guard testPruneCommandCLI can fail
[
https://issues.apache.org/jira/browse/HADOOP-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-14946:
---
Fix Version/s: 3.2.0
> S3Guard testPruneCommandCLI can fail
>
>
> Key: HADOOP-14946
> URL: https://issues.apache.org/jira/browse/HADOOP-14946
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-14946.001.patch, HADOOP-14946.002.patch,
> HADOOP-14946.003.patch
>
>
> The test of the S3Guard CLI prune can sometimes fail on parallel test runs.
> Assumption: it is the parallelism which is causing the problem
> {code}
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
> testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 10.765 sec <<< FAILURE!
> java.lang.AssertionError: Pruned children count [] expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15358) SFTPConnectionPool connections leakage
[ https://issues.apache.org/jira/browse/HADOOP-15358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HADOOP-15358: -- Assignee: Mikhail Pryakhin > SFTPConnectionPool connections leakage > -- > > Key: HADOOP-15358 > URL: https://issues.apache.org/jira/browse/HADOOP-15358 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Mikhail Pryakhin >Assignee: Mikhail Pryakhin >Priority: Critical > > Methods of SFTPFileSystem operate on poolable ChannelSftp instances, thus > some methods of SFTPFileSystem are chained together resulting in establishing > multiple connections to the SFTP server to accomplish one compound action, > those methods are listed below: > # mkdirs method > the public mkdirs method acquires a new ChannelSftp from the pool [1] > and then recursively creates directories, checking for the directory > existence beforehand by calling the method exists[2] which delegates to the > getFileStatus(ChannelSftp channel, Path file) method [3] and so on until it > ends up in returning the FilesStatus instance [4]. The resource leakage > occurs in the method getWorkingDirectory which calls the getHomeDirectory > method [5] which in turn establishes a new connection to the sftp server > instead of using an already created connection. As the mkdirs method is > recursive this results in creating a huge number of connections. > # open method [6]. This method returns an instance of FSDataInputStream > which consumes SFTPInputStream instance which doesn't return an acquired > ChannelSftp instance back to the pool but instead it closes it[7]. This leads > to establishing another connection to an SFTP server when the next method is > called on the FileSystem instance. > [1] > https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L658 > [2] > https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L321 > [3] > https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L202 > [4] > https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L290 > [5] > https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L640 > [6] > https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java#L504 > [7] > https://github.com/apache/hadoop/blob/736ceab2f58fb9ab5907c5b5110bd44384038e6b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPInputStream.java#L123 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9214) Create a new touch command to allow modifying atime and mtime
[ https://issues.apache.org/jira/browse/HADOOP-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-9214: -- Fix Version/s: 3.1.2 > Create a new touch command to allow modifying atime and mtime > - > > Key: HADOOP-9214 > URL: https://issues.apache.org/jira/browse/HADOOP-9214 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Affects Versions: 0.23.5 >Reporter: Brian Burton >Assignee: Hrishikesh Gadre >Priority: Minor > Fix For: 3.2.0, 3.1.2 > > Attachments: HADOOP-9214-001.patch, HADOOP-9214-002.patch, > HADOOP-9214-003.patch, HADOOP-9214-004.patch, HADOOP-9214-005.patch, > HADOOP-9214-006.patch > > > Currently there is no way to set the mtime or atime of a file from the > "hadoop fs" command line. It would be useful if the 'hadoop fs -touchz' > command were updated to include this functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15655) Enhance KMS client retry behavior
[
https://issues.apache.org/jira/browse/HADOOP-15655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15655:
---
Fix Version/s: 3.1.2
> Enhance KMS client retry behavior
> -
>
> Key: HADOOP-15655
> URL: https://issues.apache.org/jira/browse/HADOOP-15655
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.1.0
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Critical
> Fix For: 3.2, 3.1.2
>
> Attachments: HADOOP-15655.001.patch, HADOOP-15655.002.patch,
> HADOOP-15655.003.patch
>
>
> KMS doesn't retry upon SocketTimeoutException (the ssl connection was
> established, but the handshake timed out).
> {noformat}
> 6:08:55.315 PMWARNKMSClientProvider
> Failed to connect to example.com:16000
> 6:08:55.317 PMWARNLoadBalancingKMSClientProvider
> KMS provider at [https://example.com:16000/kms/v1/] threw an IOException:
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:140)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:473)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:472)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:788)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:288)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:124)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:949)
> at org.apache.hadoop.hbase.util.FSUtils.getVersion(FSUtils.java:338)
> at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:423)
> at
>
[jira] [Updated] (HADOOP-14212) Expose SecurityEnabled boolean field in JMX for other services besides NameNode
[
https://issues.apache.org/jira/browse/HADOOP-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-14212:
---
Fix Version/s: 3.1.2
> Expose SecurityEnabled boolean field in JMX for other services besides
> NameNode
> ---
>
> Key: HADOOP-14212
> URL: https://issues.apache.org/jira/browse/HADOOP-14212
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ray Burgemeestre
>Assignee: Adam Antal
>Priority: Minor
> Labels: security
> Fix For: 3.2.0, 3.1.2
>
> Attachments: HADOOP-14212.001.patch, HADOOP-14212.002.patch,
> HADOOP-14212.003.patch, HADOOP-14212.004.patch, HADOOP-14212.005.patch,
> HADOOP-14212.005.patch, HADOOP-14212.005.patch, HADOOP-14212.006.patch,
> HADOOP-14212.007.patch, HADOOP-14212.008.patch, HADOOP-14212.009.patch
>
>
> The following commit
> https://github.com/apache/hadoop/commit/dc17bda4b677e30c02c2a9a053895a43e41f7a12
> introduced a "SecurityEnabled" field in the JMX output for the NameNode. I
> believe it would be nice to add this same change to the JMX output of other
> services: Secondary Namenode, ResourceManager, NodeManagers, DataNodes, etc.
> So that it can be queried whether Security is enabled in all JMX resources.
> The reason I am suggesting this feature / improvement is that I think it
> would provide a clean way to check whether your cluster is completely
> Kerberized or not. I don't think there is an easy/clean way to do this now,
> other than checking the logs, checking ports etc.?
> The file where the change was made is
> hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> has the following function now:
> {code:java}
> @Override // NameNodeStatusMXBean
> public boolean isSecurityEnabled() {
> return UserGroupInformation.isSecurityEnabled();
> }
> {code}
> I would be happy to develop a patch if it seems useful by others as well?
> This is a snippet from the JMX output from the NameNode in case security is
> not enabled:
> {code}
> {
> "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
> "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
> "NNRole" : "NameNode",
> "HostAndPort" : "node001.cm.cluster:8020",
> "SecurityEnabled" : false,
> "LastHATransitionTime" : 0,
> "State" : "standby"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15476) fix logging for split-dns multihome
[ https://issues.apache.org/jira/browse/HADOOP-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15476: --- Issue Type: Improvement (was: Bug) > fix logging for split-dns multihome > > > Key: HADOOP-15476 > URL: https://issues.apache.org/jira/browse/HADOOP-15476 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HADOOP-15476-branch-3.1.00.patch, HADOOP-15476.00.patch, > HADOOP-15476.01.patch > > > Fix debug log statement introduced in [HADOOP-15250]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15476) fix logging for split-dns multihome
[ https://issues.apache.org/jira/browse/HADOOP-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15476: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1. I've committed this. Thanks [~ajayydv]. > fix logging for split-dns multihome > > > Key: HADOOP-15476 > URL: https://issues.apache.org/jira/browse/HADOOP-15476 > Project: Hadoop Common > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: HADOOP-15476-branch-3.1.00.patch, HADOOP-15476.00.patch, > HADOOP-15476.01.patch > > > Fix debug log statement introduced in [HADOOP-15250]. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15493) DiskChecker should handle disk full situation
[
https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530413#comment-16530413
]
Arpit Agarwal edited comment on HADOOP-15493 at 7/2/18 8:36 PM:
{quote}I think we have to rely on the system to detect a failed
controller/drive. Maybe we should just attempt to provoke the disk to go
read-only. Have the DN periodically write a file to its storages every n-many
mins – but take no action upon failure. Instead rely on the normal disk check
to subsequently discover the disk is read-only.
{quote}
When you say 'we have to rely on the system', do you mean the OS?
We saw disk failures (and rarely controller failures) go undetected
indefinitely. Application requests would fail and trigger disk checker which
always succeeded. We had customers hit data loss after multiple undetected disk
failures over a few days.
{quote}I don't think this disk-is-writable check should be in common.
{quote}
We can make the write check HDFS-internal. We still need a disk full check if
the write fails. Perhaps the safest option is a threshold which avoids false
positives and allows false negatives.
was (Author: arpitagarwal):
{quote}I think we have to rely on the system to detect a failed
controller/drive. Maybe we should just attempt to provoke the disk to go
read-only. Have the DN periodically write a file to its storages every n-many
mins – but take no action upon failure. Instead rely on the normal disk check
to subsequently discover the disk is read-only.
{quote}
When you say 'we have to rely on the system', do you mean the OS?
We saw disk failures (and rarely controller failures) go undetected
indefinitely. Application requests would fail and trigger disk checker which
always succeeded. We had customers hit data loss after multiple undetected disk
failures over a few days.
{quote}I don't think this disk-is-writable check should be in common.
{quote}
We can make the write check HDFS-internal. We still need a disk full check.
Perhaps the safest option is a threshold which avoids false positives and
allows false negatives.
> DiskChecker should handle disk full situation
> -
>
> Key: HADOOP-15493
> URL: https://issues.apache.org/jira/browse/HADOOP-15493
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Critical
> Attachments: HADOOP-15493.01.patch, HADOOP-15493.02.patch
>
>
> DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is
> writable.
> However check should not fail when file creation fails due to disk being
> full. This avoids marking full disks as _failed_.
> Reported by [~kihwal] and [~daryn] in HADOOP-15450.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15493) DiskChecker should handle disk full situation
[
https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530413#comment-16530413
]
Arpit Agarwal commented on HADOOP-15493:
{quote}I think we have to rely on the system to detect a failed
controller/drive. Maybe we should just attempt to provoke the disk to go
read-only. Have the DN periodically write a file to its storages every n-many
mins – but take no action upon failure. Instead rely on the normal disk check
to subsequently discover the disk is read-only.
{quote}
When you say 'we have to rely on the system', do you mean the OS?
We saw disk failures (and rarely controller failures) go undetected
indefinitely. Application requests would fail and trigger disk checker which
always succeeded. We had customers hit data loss after multiple undetected disk
failures over a few days.
{quote}I don't think this disk-is-writable check should be in common.
{quote}
We can make the write check HDFS-internal. We still need a disk full check.
Perhaps the safest option is a threshold which avoids false positives and
allows false negatives.
> DiskChecker should handle disk full situation
> -
>
> Key: HADOOP-15493
> URL: https://issues.apache.org/jira/browse/HADOOP-15493
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Critical
> Attachments: HADOOP-15493.01.patch, HADOOP-15493.02.patch
>
>
> DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is
> writable.
> However check should not fail when file creation fails due to disk being
> full. This avoids marking full disks as _failed_.
> Reported by [~kihwal] and [~daryn] in HADOOP-15450.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15574) Suppress build error if there are no docs after excluding private annotations
[
https://issues.apache.org/jira/browse/HADOOP-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527901#comment-16527901
]
Arpit Agarwal edited comment on HADOOP-15574 at 6/29/18 4:25 PM:
-
Moved this to the Hadoop Project.
I'll take a look although I am not qualified to comment on this part of the
code.
was (Author: arpitagarwal):
This should be moved to the Hadoop Project.
I'll take a look although I am not qualified to comment on this part of the
code.
> Suppress build error if there are no docs after excluding private annotations
> -
>
> Key: HADOOP-15574
> URL: https://issues.apache.org/jira/browse/HADOOP-15574
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDDS-202.1.patch
>
>
> Seen in hadoop-ozone when building with the Maven hdds profile enabled.
> {noformat}
> $ mvn clean install -DskipTests -DskipShade -Phdds -Pdist --projects
> hadoop-ozone/ozonefs
> ...
> [INFO] --- maven-javadoc-plugin:3.0.0-M1:jar (module-javadocs) @
> hadoop-ozone-filesystem ---
> [INFO]
> ExcludePrivateAnnotationsStandardDoclet
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 13.223 s
> [INFO] Finished at: 2018-06-28T19:46:49+09:00
> [INFO] Final Memory: 122M/1196M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-javadoc-plugin:3.0.0-M1:jar (module-javadocs)
> on project hadoop-ozone-filesystem: MavenReportException: Error while
> generating Javadoc:
> [ERROR] Exit code: 1 - Picked up _JAVA_OPTIONS: -Duser.language=en
> [ERROR] java.lang.ArrayIndexOutOfBoundsException: 0
> [ERROR] at
> com.sun.tools.doclets.formats.html.ConfigurationImpl.setTopFile(ConfigurationImpl.java:537)
> [ERROR] at
> com.sun.tools.doclets.formats.html.ConfigurationImpl.setSpecificDocletOptions(ConfigurationImpl.java:309)
> [ERROR] at
> com.sun.tools.doclets.internal.toolkit.Configuration.setOptions(Configuration.java:560)
> [ERROR] at
> com.sun.tools.doclets.internal.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:134)
> [ERROR] at
> com.sun.tools.doclets.internal.toolkit.AbstractDoclet.start(AbstractDoclet.java:82)
> [ERROR] at
> com.sun.tools.doclets.formats.html.HtmlDoclet.start(HtmlDoclet.java:80)
> [ERROR] at
> com.sun.tools.doclets.standard.Standard.start(Standard.java:39)
> [ERROR] at
> org.apache.hadoop.classification.tools.ExcludePrivateAnnotationsStandardDoclet.start(ExcludePrivateAnnotationsStandardDoclet.java:41)
> [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [ERROR] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
> [ERROR] at
> com.sun.tools.javadoc.DocletInvoker.invoke(DocletInvoker.java:310)
> [ERROR] at
> com.sun.tools.javadoc.DocletInvoker.start(DocletInvoker.java:189)
> [ERROR] at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:366)
> [ERROR] at com.sun.tools.javadoc.Start.begin(Start.java:219)
> [ERROR] at com.sun.tools.javadoc.Start.begin(Start.java:205)
> [ERROR] at com.sun.tools.javadoc.Main.execute(Main.java:64)
> [ERROR] at com.sun.tools.javadoc.Main.main(Main.java:54)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-15574) Suppress build error if there are no docs after excluding private annotations
[
https://issues.apache.org/jira/browse/HADOOP-15574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal moved HDDS-202 to HADOOP-15574:
-
Target Version/s: 3.2.0, 3.1.1
Component/s: (was: Ozone Filesystem)
Workflow: no-reopen-closed, patch-avail (was: patch-available,
re-open possible)
Key: HADOOP-15574 (was: HDDS-202)
Project: Hadoop Common (was: Hadoop Distributed Data Store)
> Suppress build error if there are no docs after excluding private annotations
> -
>
> Key: HADOOP-15574
> URL: https://issues.apache.org/jira/browse/HADOOP-15574
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDDS-202.1.patch
>
>
> Seen in hadoop-ozone when building with the Maven hdds profile enabled.
> {noformat}
> $ mvn clean install -DskipTests -DskipShade -Phdds -Pdist --projects
> hadoop-ozone/ozonefs
> ...
> [INFO] --- maven-javadoc-plugin:3.0.0-M1:jar (module-javadocs) @
> hadoop-ozone-filesystem ---
> [INFO]
> ExcludePrivateAnnotationsStandardDoclet
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 13.223 s
> [INFO] Finished at: 2018-06-28T19:46:49+09:00
> [INFO] Final Memory: 122M/1196M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-javadoc-plugin:3.0.0-M1:jar (module-javadocs)
> on project hadoop-ozone-filesystem: MavenReportException: Error while
> generating Javadoc:
> [ERROR] Exit code: 1 - Picked up _JAVA_OPTIONS: -Duser.language=en
> [ERROR] java.lang.ArrayIndexOutOfBoundsException: 0
> [ERROR] at
> com.sun.tools.doclets.formats.html.ConfigurationImpl.setTopFile(ConfigurationImpl.java:537)
> [ERROR] at
> com.sun.tools.doclets.formats.html.ConfigurationImpl.setSpecificDocletOptions(ConfigurationImpl.java:309)
> [ERROR] at
> com.sun.tools.doclets.internal.toolkit.Configuration.setOptions(Configuration.java:560)
> [ERROR] at
> com.sun.tools.doclets.internal.toolkit.AbstractDoclet.startGeneration(AbstractDoclet.java:134)
> [ERROR] at
> com.sun.tools.doclets.internal.toolkit.AbstractDoclet.start(AbstractDoclet.java:82)
> [ERROR] at
> com.sun.tools.doclets.formats.html.HtmlDoclet.start(HtmlDoclet.java:80)
> [ERROR] at
> com.sun.tools.doclets.standard.Standard.start(Standard.java:39)
> [ERROR] at
> org.apache.hadoop.classification.tools.ExcludePrivateAnnotationsStandardDoclet.start(ExcludePrivateAnnotationsStandardDoclet.java:41)
> [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [ERROR] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
> [ERROR] at
> com.sun.tools.javadoc.DocletInvoker.invoke(DocletInvoker.java:310)
> [ERROR] at
> com.sun.tools.javadoc.DocletInvoker.start(DocletInvoker.java:189)
> [ERROR] at com.sun.tools.javadoc.Start.parseAndExecute(Start.java:366)
> [ERROR] at com.sun.tools.javadoc.Start.begin(Start.java:219)
> [ERROR] at com.sun.tools.javadoc.Start.begin(Start.java:205)
> [ERROR] at com.sun.tools.javadoc.Main.execute(Main.java:64)
> [ERROR] at com.sun.tools.javadoc.Main.main(Main.java:54)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15074) SequenceFile#Writer flush does not update the length of the written file.
[
https://issues.apache.org/jira/browse/HADOOP-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523129#comment-16523129
]
Arpit Agarwal commented on HADOOP-15074:
Never mind, I think I understand what you mean now. There is no way to make
this change without exposing update_length flag via FileSystem.
> SequenceFile#Writer flush does not update the length of the written file.
> -
>
> Key: HADOOP-15074
> URL: https://issues.apache.org/jira/browse/HADOOP-15074
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
>
> SequenceFile#Writer flush does not update the length of the file. This
> happens because as part of the flush, {{UPDATE_LENGTH}} flag is not passed to
> the DFSOutputStream#hsync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15074) SequenceFile#Writer flush does not update the length of the written file.
[
https://issues.apache.org/jira/browse/HADOOP-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522778#comment-16522778
]
Arpit Agarwal commented on HADOOP-15074:
bq. I don't like that, HdfsDataOutputStream is in hdfs-client JAR; you can't
loop in hadoop-common.
[~ste...@apache.org], sorry I didn't get what you mean. The change I suggested
doesn't need any jar dependency changes.
> SequenceFile#Writer flush does not update the length of the written file.
> -
>
> Key: HADOOP-15074
> URL: https://issues.apache.org/jira/browse/HADOOP-15074
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
>
> SequenceFile#Writer flush does not update the length of the file. This
> happens because as part of the flush, {{UPDATE_LENGTH}} flag is not passed to
> the DFSOutputStream#hsync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15074) SequenceFile#Writer flush does not update the length of the written file.
[
https://issues.apache.org/jira/browse/HADOOP-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519699#comment-16519699
]
Arpit Agarwal commented on HADOOP-15074:
Hi [~harishjp], can you check whether the returned {{FSDataOutputStream}} is an
instance of {{HdfsDataOutputStream}}. If so, then you can invoke
_hsync(EnumSet syncFlags)_ on it. HdfsDataOutputStream is a public
interface.
I don't usually like suggesting reflection, but in this case it may be the
cleanest solution.
> SequenceFile#Writer flush does not update the length of the written file.
> -
>
> Key: HADOOP-15074
> URL: https://issues.apache.org/jira/browse/HADOOP-15074
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
>
> SequenceFile#Writer flush does not update the length of the file. This
> happens because as part of the flush, {{UPDATE_LENGTH}} flag is not passed to
> the DFSOutputStream#hsync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15493) DiskChecker should handle disk full situation
[
https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502139#comment-16502139
]
Arpit Agarwal commented on HADOOP-15493:
Yeah checking for the messages is terribly ugly. However I don't know any other
filesystem agnostic way of checking for disk full situation. Threshold-based
checks are inaccurate:
# Space may be freed up between the failure and running {{getUsableSpace}}.
# Filesystems may not report zero free space when the disk is full (especially
xfs) so we can't compare with zero. We'll need an inaccurate threshold.
> DiskChecker should handle disk full situation
> -
>
> Key: HADOOP-15493
> URL: https://issues.apache.org/jira/browse/HADOOP-15493
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Critical
> Attachments: HADOOP-15493.01.patch, HADOOP-15493.02.patch
>
>
> DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is
> writable.
> However check should not fail when file creation fails due to disk being
> full. This avoids marking full disks as _failed_.
> Reported by [~kihwal] and [~daryn] in HADOOP-15450.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15446) WASB: PageBlobInputStream.skip breaks HBASE replication
[ https://issues.apache.org/jira/browse/HADOOP-15446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15446: --- Fix Version/s: 3.1.1 > WASB: PageBlobInputStream.skip breaks HBASE replication > --- > > Key: HADOOP-15446 > URL: https://issues.apache.org/jira/browse/HADOOP-15446 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 2.9.0, 3.0.2 >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.1.1 > > Attachments: HADOOP-15446-001.patch, HADOOP-15446-002.patch, > HADOOP-15446-003.patch, HADOOP-15446-branch-2.001.patch > > > Page Blobs are primarily used by HBASE. HBASE replication, which apparently > has not been used with WASB until recently, performs non-sequential reads on > log files using PageBlobInputStream. There are bugs in this stream > implementation which prevent skip and seek from working properly, and > eventually the stream state becomes corrupt and unusable. > I believe this bug affects all releases of WASB/HADOOP. It appears to be a > day-0 bug in PageBlobInputStream. There were similar bugs opened in the past > (HADOOP-15042) but the issue was not properly fixed, and no test coverage was > added. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15499) Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder
[ https://issues.apache.org/jira/browse/HADOOP-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15499: --- Affects Version/s: 3.1.1 > Performance severe drop when running RawErasureCoderBenchmark with > NativeRSRawErasureCoder > -- > > Key: HADOOP-15499 > URL: https://issues.apache.org/jira/browse/HADOOP-15499 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.0, 3.0.1, 3.0.2, 3.1.1 >Reporter: SammiChen >Assignee: SammiChen >Priority: Major > Attachments: HADOOP-15499.001.patch > > > Run RawErasureCoderBenchmark which is a micro-benchmark to test EC codec > encoding/decoding performance. > 50 concurrency Native ISA-L coder has the less throughput than 1 concurrency > Native ISA-L case. It's abnormal. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 1 > 1024 1024 > Using 126MB buffer. > ISA-L coder encode 1008MB data, with chunk size 1024KB > Total time: 0.19 s. > Total throughput: 5390.37 MB/s > Threads statistics: > 1 threads in total. > Min: 0.18 s, Max: 0.18 s, Avg: 0.18 s, 90th Percentile: 0.18 s. > > bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar > org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 > 50 1024 10240 > Using 120MB buffer. > ISA-L coder encode 54000MB data, with chunk size 10240KB > Total time: 11.58 s. > Total throughput: 4662 MB/s > Threads statistics: > 50 threads in total. > Min: 0.55 s, Max: 11.5 s, Avg: 6.32 s, 90th Percentile: 10.45 s. > > RawErasureCoderBenchmark shares a single coder between all concurrent > threads. While > NativeRSRawEncoder and NativeRSRawDecoder has synchronized key work on > doDecode and doEncode function. So 50 concurrent threads are forced to use > the shared coder encode/decode function one by one. > > To resolve the issue, there are two approaches. > # Refactor RawErasureCoderBenchmark to use dedicated coder for each > concurrent thread. > # Refactor NativeRSRawEncoder and NativeRSRawDecoder to get better > concurrency. Since the synchronized key work is to try to protect the > private variable nativeCoder from being checked in doEncode/doDecode and > being modified in release. We can use reentrantReadWriteLock to increase the > concurrency since doEncode/doDecode can be called multiple times without > change the nativeCoder state. > I prefer approach 2 and will upload a patch later. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15493) DiskChecker should handle disk full situation
[ https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15493: --- Attachment: HADOOP-15493.02.patch > DiskChecker should handle disk full situation > - > > Key: HADOOP-15493 > URL: https://issues.apache.org/jira/browse/HADOOP-15493 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Critical > Attachments: HADOOP-15493.01.patch, HADOOP-15493.02.patch > > > DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is > writable. > However check should not fail when file creation fails due to disk being > full. This avoids marking full disks as _failed_. > Reported by [~kihwal] and [~daryn] in HADOOP-15450. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15476) fix logging for split-dns multihome
[
https://issues.apache.org/jira/browse/HADOOP-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489914#comment-16489914
]
Arpit Agarwal commented on HADOOP-15476:
Thanks for catching this [~ajayydv].
Can we just remove this log message? There is another message below:
{code}
LOG.debug("Binding {} to {}", principal, localAddr);
{code}
For wildcard address, localAddr will be null. So perhaps we can fix this other
log message to print wildcard if localAddr is null.
> fix logging for split-dns multihome
>
>
> Key: HADOOP-15476
> URL: https://issues.apache.org/jira/browse/HADOOP-15476
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HADOOP-15476-branch-3.1.00.patch, HADOOP-15476.00.patch
>
>
> Fix debug log statement introduced in [HADOOP-15250].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15493) DiskChecker should handle disk full situation
[ https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489910#comment-16489910 ] Arpit Agarwal commented on HADOOP-15493: The patch looks for disk full as follows: # Checks exception message for known strings seen on Linux/glibc and Windows. # Checks for usable disk space below threshold (currently a conservative 100MB). We need a threshold as filesystems don't always report zero usable space when the disk is full. If either condition is true, then failure to write the file is ignored. Note: no one is using this disk check yet. It will be enabled later via HDFS-13538. [~kihwal], could you please take a look? > DiskChecker should handle disk full situation > - > > Key: HADOOP-15493 > URL: https://issues.apache.org/jira/browse/HADOOP-15493 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Critical > Attachments: HADOOP-15493.01.patch > > > DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is > writable. > However check should not fail when file creation fails due to disk being > full. This avoids marking full disks as _failed_. > Reported by [~kihwal] and [~daryn] in HADOOP-15450. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15493) DiskChecker should handle disk full situation
[ https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15493: --- Status: Patch Available (was: Open) > DiskChecker should handle disk full situation > - > > Key: HADOOP-15493 > URL: https://issues.apache.org/jira/browse/HADOOP-15493 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Critical > Attachments: HADOOP-15493.01.patch > > > DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is > writable. > However check should not fail when file creation fails due to disk being > full. This avoids marking full disks as _failed_. > Reported by [~kihwal] and [~daryn] in HADOOP-15450. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15493) DiskChecker should handle disk full situation
[ https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HADOOP-15493: --- Attachment: HADOOP-15493.01.patch > DiskChecker should handle disk full situation > - > > Key: HADOOP-15493 > URL: https://issues.apache.org/jira/browse/HADOOP-15493 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Critical > Attachments: HADOOP-15493.01.patch > > > DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is > writable. > However check should not fail when file creation fails due to disk being > full. This avoids marking full disks as _failed_. > Reported by [~kihwal] and [~daryn] in HADOOP-15450. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15493) DiskChecker should handle disk full situation
Arpit Agarwal created HADOOP-15493: -- Summary: DiskChecker should handle disk full situation Key: HADOOP-15493 URL: https://issues.apache.org/jira/browse/HADOOP-15493 Project: Hadoop Common Issue Type: Improvement Reporter: Arpit Agarwal Assignee: Arpit Agarwal DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is writable. However check should not fail when file creation fails due to disk being full. This avoids marking full disks as _failed_. Reported by [~kihwal] and [~daryn] in HADOOP-15450. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15486) Make NetworkTopology#netLock fair
[
https://issues.apache.org/jira/browse/HADOOP-15486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15486:
---
Fix Version/s: 3.0.3
> Make NetworkTopology#netLock fair
> -
>
> Key: HADOOP-15486
> URL: https://issues.apache.org/jira/browse/HADOOP-15486
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.3, 2.7.7, 2.8.5
>
> Attachments: HADOOP-15486.000.patch, HADOOP-15486.001.patch
>
>
> Whenever a datanode is restarted, the registration call after the restart
> received by NameNode lands in {{NetworkTopology#add}} via
> {{DatanodeManager#registerDatanode}} requires write lock on
> {{NetworkTopology#netLock}}.
> This registration thread is getting starved by flood of
> {{FSNamesystem.getAdditionalDatanode}} calls, which are triggered by clients
> those who were writing to the restarted datanode.
> The registration call which is waiting for write lock on
> {{NetworkTopology#netLock}} is holding write lock on {{FSNamesystem#fsLock}},
> causing all the other RPC calls which require the lock on
> {{FSNamesystem#fsLock}} wait.
> We can make {{NetworkTopology#netLock}} lock fair so that the registration
> thread will not starve.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487881#comment-16487881 ] Arpit Agarwal commented on HADOOP-15483: [~ljain] thanks for this improvement. Do we also need to update the references in the pom files? e.g. https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml#L399 > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15483.001.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
