[GitHub] [hadoop-ozone] captainzmc removed a comment on pull request #1280: HDDS-4054. Remove duplicate judgments in allocateBlockInKey.
captainzmc removed a comment on pull request #1280: URL: https://github.com/apache/hadoop-ozone/pull/1280#issuecomment-667041335 Hi @bharatviswa504 @arp7 Could you help to review this？ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runitao commented on pull request #1289: HDDS-4039. Reduce the number of fields in hdds.proto to improve performance
runitao commented on pull request #1289: URL: https://github.com/apache/hadoop-ozone/pull/1289#issuecomment-668919240 /pending This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4039) Reduce the number of fields in hdds.proto to improve performance
[ https://issues.apache.org/jira/browse/HDDS-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4039: - Labels: pull-request-available (was: ) > Reduce the number of fields in hdds.proto to improve performance > > > Key: HDDS-4039 > URL: https://issues.apache.org/jira/browse/HDDS-4039 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Datanode >Affects Versions: 0.6.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: HuangTao >Priority: Major > Labels: pull-request-available > > HDDS-3989 introduced revision and buildDate fields to hdds.proto file. These > fields are required only for Recon UI and don't have to be part of hdds.proto. > Also, version and setupTime are other two fields which can be removed and > added only to the SCM registration type as per [~elek] > ([https://github.com/apache/hadoop-ozone/pull/1226|https://github.com/apache/hadoop-ozone/pull/1226#issuecomment-663416483]) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runitao opened a new pull request #1289: HDDS-4039. Reduce the number of fields in hdds.proto to improve performance
runitao opened a new pull request #1289: URL: https://github.com/apache/hadoop-ozone/pull/1289 ## What changes were proposed in this pull request? Reduce the number of fields in hdds.proto to improve performance ## What is the link to the Apache JIRA HDDS-4039 ## How was this patch tested? Test with docker-compose development enviroment. And test by CI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4063) Fix InstallSnapshot in OM HA
Hanisha Koneru created HDDS-4063: Summary: Fix InstallSnapshot in OM HA Key: HDDS-4063 URL: https://issues.apache.org/jira/browse/HDDS-4063 Project: Hadoop Distributed Data Store Issue Type: Bug Components: OM HA Reporter: Hanisha Koneru Assignee: Hanisha Koneru OzoneManagerStateMachine#notifyInstallSnapshotFromLeader() checks the incoming roleInfoProto and proceeds with install snapshot request only if the role is Leader. This check is wrong and the roleInfoProto will contain the self node ID and not the leaders. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4040) OFS should use deleteObjects when deleting directory
[ https://issues.apache.org/jira/browse/HDDS-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171086#comment-17171086 ] Siyao Meng commented on HDDS-4040: -- [~Sammi] Should be good. I have posted PR. Thanks for asking :D > OFS should use deleteObjects when deleting directory > > > Key: HDDS-4040 > URL: https://issues.apache.org/jira/browse/HDDS-4040 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Bharat Viswanadham >Assignee: Siyao Meng >Priority: Blocker > Labels: pull-request-available > > This Jira is to use deleteObjects in OFS delete now that HDDS-3286 is > committed. > Currently when ozone.om.enable.filesystem.paths is enabled it normalizes the > path, so using deleteKey for delete directory will fail. > According to [~bharat] this should be a blocker for 0.6.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1745) Add integration test for createDirectory for OM HA
[ https://issues.apache.org/jira/browse/HDDS-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17171031#comment-17171031 ] Rui Wang commented on HDDS-1745: In fact, for 1 I find the model in https://issues.apache.org/jira/browse/HDDS-505 also cc [~hanishakoneru] in case you know the answer to 2. > Add integration test for createDirectory for OM HA > -- > > Key: HDDS-1745 > URL: https://issues.apache.org/jira/browse/HDDS-1745 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA, Ozone Manager >Reporter: Bharat Viswanadham >Priority: Major > Labels: newbie > > Add an integration test for createDirectory which is implemented as part of > HDDS-1730 for OM HA. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] arp7 commented on pull request #1270: HDDS-4044. Deprecate ozone.s3g.volume.name.
arp7 commented on pull request #1270: URL: https://github.com/apache/hadoop-ozone/pull/1270#issuecomment-668741765 Leaving this key easily discoverable is a bad idea and will result in users unintentionally shooting themselves in the foot. However I don't have the energy to argue this ad-infinitum so in the interests of making forward progress, let's just go ahead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1270: HDDS-4044. Deprecate ozone.s3g.volume.name.
bharatviswa504 commented on pull request #1270: URL: https://github.com/apache/hadoop-ozone/pull/1270#issuecomment-668733523 Removed all the code changes, in the latest commit updated only documentation. Thank You @elek for the review and @arp7 for offline discussion and consensus. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4062) Non rack aware pipelines should not be created if multiple racks are alive
Stephen O'Donnell created HDDS-4062: --- Summary: Non rack aware pipelines should not be created if multiple racks are alive Key: HDDS-4062 URL: https://issues.apache.org/jira/browse/HDDS-4062 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Affects Versions: 0.7.0 Reporter: Stephen O'Donnell Assignee: Stephen O'Donnell If we have a scenario where one rack has more nodes that others, it is possible for all hosts in the cluster to have reached their pipeline limit, while 3 nodes on the larger rack have not. The current fallback logic will then allow a pipeline to be created which uses only the 3 nodes on the same rack, violating the rack placement policy. There may be other ways this could happen with cluster load too, were the pipeline capacity has reached its limit on some nodes but not others. The proposal here, is that if the cluster has multiple racks AND there are healthy nodes covering at least 2 racks, where healthy is defined as a node which is registered and not stale or dead, then we should not allow "fallback" (pipelines which span only 1 rack) pipelines to be created. This means if you have a badly configured cluster - eg Rack 1 = 10 nodes; Rack 2 = 1 node, the pipeline limit will be constrained by the capacity of that 1 node on rack 2. Even a setup like Rack 1 = 10 nodes, Rack 2 = 5 would be constrained by this. This constraint is better than creating non rack aware pipelines, and the rule above will handle the case when the cluster degrades to 1 rack, as the healthy node definition will notice only 1 rack is alive. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant commented on pull request #1154: HDDS-3867. Extend the chunkinfo tool to display information from all nodes in the pipeline.
bshashikant commented on pull request #1154: URL: https://github.com/apache/hadoop-ozone/pull/1154#issuecomment-668696169 > > Thanks to create this patch @sadanand48 > > I have one design concern related to this change. I am not sure what is the good answer, but trying to share my view: > > This patch modifies the `client` interface to provide more functionality for a `debug` tool. I like this more functionality, but I think we shouldn't add any debug specific feature to the main client API. Client API should be backward compatible between releases and it might have multiple implementations. > > I would prefer to separate the `client` api from the `admin` api (used by this tool). > > As an example: Let's say I would like to implement a new client for Erasure Coding. If the client interface contains debug specific methods I need to implement all of them, which makes introduce any new feature harder. > > I am not sure how is it possible to keep this separation (I need your feedback). One possible solution is ot move the logic of looping over the datanodes to the `ChunkKeyHandler`. But it's not clear how can it work with RATIS pipeline. Or do you use STANDALONE client to read the data which is persisted with RATIS? > > I think we should remove XceiverClient interface altogether. Read and write path client functionality seem to diverge a lot . > Adding a new functionality in XceiverClientGrpc will just be fine then. > > There is alredy a jira filed for the same https://issues.apache.org/jira/browse/HDDS-998. @elek , I feel if we can go ahead with this change and then try to remove XceiverClient interface altogether in a separte jira. Your thoughts?? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] errose28 opened a new pull request #1288: HDDS-4061. Pending delete blocks are not always included in #BLOCKCOUNT metadata
errose28 opened a new pull request #1288: URL: https://github.com/apache/hadoop-ozone/pull/1288 ## What changes were proposed in this pull request? - Make `KeyValueContainerUtil#initializeUsedBytesAndBlockCount` include pending delete blocks (with `#deleting#` prefix) when setting the `#BLOCKCOUNT` and `#BYTESUSED` metadata values. - This behavior makes it consistent with the current `BlockDeletingService` implementation. - Adjust the TestContainerReader unit tests to account for this change. ## What is the link to the Apache JIRA HDDS-4061 ## How was this patch tested? `TestContainerReader#testContainerReader` unit tests were modified to include pending delete blocks in the `#BLOCKCOUNT` and `#BYTESUSED` metadata values. These tests were run and passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4061) Pending delete blocks are not always included in #BLOCKCOUNT metadata
[ https://issues.apache.org/jira/browse/HDDS-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4061: - Labels: pull-request-available (was: ) > Pending delete blocks are not always included in #BLOCKCOUNT metadata > - > > Key: HDDS-4061 > URL: https://issues.apache.org/jira/browse/HDDS-4061 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Minor > Labels: pull-request-available > > Within the code that handles containers, there is currently some disagreement > as to whether blocks marked for deletion with the #deleting# prefix should be > included in the #BLOCKCOUNT metadata value. The BlockDeletingService includes > pending delete blocks in the block count, since after deleting the blocks, > BlockDeletingService#call calls > KeyValueContainerData#updateAndCommitDBCounters, where the pending deletion > blocks (now deleted) are subtracted from the block count metadata. However, > KeyValueContainerUtil#initializeUsedBytesAndBlockCount filters away all > blocks that have prefixes when setting the block count. This means pending > delete blocks with the #deleting# prefix are excluded from the block count. > This fix will alter KeyValueContainerUtil#initializeUsedBytesAndBlockCount to > include #deleting# blocks when setting the #BLOCKCOUNT and #BYTESUSED > metadata values. It will also adjust the TestContainerReader unit tests to > account for this change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4061) Pending delete blocks are not always included in #BLOCKCOUNT metadata
Ethan Rose created HDDS-4061: Summary: Pending delete blocks are not always included in #BLOCKCOUNT metadata Key: HDDS-4061 URL: https://issues.apache.org/jira/browse/HDDS-4061 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Datanode Reporter: Ethan Rose Assignee: Ethan Rose Within the code that handles containers, there is currently some disagreement as to whether blocks marked for deletion with the #deleting# prefix should be included in the #BLOCKCOUNT metadata value. The BlockDeletingService includes pending delete blocks in the block count, since after deleting the blocks, BlockDeletingService#call calls KeyValueContainerData#updateAndCommitDBCounters, where the pending deletion blocks (now deleted) are subtracted from the block count metadata. However, KeyValueContainerUtil#initializeUsedBytesAndBlockCount filters away all blocks that have prefixes when setting the block count. This means pending delete blocks with the #deleting# prefix are excluded from the block count. This fix will alter KeyValueContainerUtil#initializeUsedBytesAndBlockCount to include #deleting# blocks when setting the #BLOCKCOUNT and #BYTESUSED metadata values. It will also adjust the TestContainerReader unit tests to account for this change. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1149: HDDS-3878. Make OMHA serviceID optional if one (but only one) is defined in the config
elek commented on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-668600956 @arp7 @bharatviswa504 Are you OK with this approach? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1889) Add support for verifying multiline log entry
[
https://issues.apache.org/jira/browse/HDDS-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Orova reassigned HDDS-1889:
-
Assignee: Peter Orova
> Add support for verifying multiline log entry
> -
>
> Key: HDDS-1889
> URL: https://issues.apache.org/jira/browse/HDDS-1889
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Reporter: Dinesh Chitlangia
>Assignee: Peter Orova
>Priority: Major
> Labels: newbie
> Attachments: image.png
>
>
> This jira aims to test the failure scenario where a multi-line stack trace
> will be added in the audit log. Currently, for test assumes that even in
> failure scenario we don't have multi-line log entry.
> Example:
> {code:java}
> private static final AuditMessage READ_FAIL_MSG =
> new AuditMessage.Builder()
> .setUser("john")
> .atIp("192.168.0.1")
> .forOperation(DummyAction.READ_VOLUME.name())
> .withParams(PARAMS)
> .withResult(FAILURE)
> .withException(null).build();
> {code}
> Therefore in verifyLog() we only compare for first line of the log file with
> the expected message.
> The test would fail if in future someone were to create a scenario with
> multi-line log entry.
> 1. Update READ_FAIL_MSG so that it has multiple lines of Exception stack
> trace.
> This is what multi-line log entry could look like:
> {code:java}
> ERROR | OMAudit | user=dchitlangia | ip=127.0.0.1 | op=GET_ACL
> {volume=volume80100, bucket=bucket83878, key=null, aclType=CREATE,
> resourceType=volume, storeType=ozone} | ret=FAILURE
> org.apache.hadoop.ozone.om.exceptions.OMException: User dchitlangia doesn't
> have CREATE permission to access volume
> at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1809)
> ~[classes/:?]
> at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1769)
> ~[classes/:?]
> at
> org.apache.hadoop.ozone.om.OzoneManager.createBucket(OzoneManager.java:2092)
> ~[classes/:?]
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.createBucket(OzoneManagerRequestHandler.java:526)
> ~[classes/:?]
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handle(OzoneManagerRequestHandler.java:185)
> ~[classes/:?]
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:192)
> ~[classes/:?]
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:110)
> ~[classes/:?]
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> ~[classes/:?]
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> ~[hadoop-common-3.2.0.jar:?]
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> ~[hadoop-common-3.2.0.jar:?]
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> ~[hadoop-common-3.2.0.jar:?]
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> ~[hadoop-common-3.2.0.jar:?]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_144]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> ~[hadoop-common-3.2.0.jar:?]
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> ~[hadoop-common-3.2.0.jar:?]
> {code}
> 2. Update verifyLog method to accept variable number of arguments.
> 3. Update the assertion so that it compares beyond the first line when the
> expected is a multi-line log entry.
> {code:java}
> assertTrue(expected.equalsIgnoreCase(lines.get(0)));
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3261) Fix TestOzoneManagerRestart.java
[ https://issues.apache.org/jira/browse/HDDS-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170755#comment-17170755 ] Attila Doroszlai commented on HDDS-3261: Thanks [~orova] for checking. Yes, it seems to be dup of another issue just fixed today. > Fix TestOzoneManagerRestart.java > > > Key: HDDS-3261 > URL: https://issues.apache.org/jira/browse/HDDS-3261 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Shashikant Banerjee >Assignee: Peter Orova >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3261) Fix TestOzoneManagerRestart.java
[ https://issues.apache.org/jira/browse/HDDS-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170749#comment-17170749 ] Peter Orova commented on HDDS-3261: --- [~adoroszlai] checked this one against ff621c6f0083f67862a5fbb554f13761ea97113b did not manage to get a failure. Is this a duplicate of a fixed jira perhaps? > Fix TestOzoneManagerRestart.java > > > Key: HDDS-3261 > URL: https://issues.apache.org/jira/browse/HDDS-3261 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Shashikant Banerjee >Assignee: Peter Orova >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3261) Fix TestOzoneManagerRestart.java
[ https://issues.apache.org/jira/browse/HDDS-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Orova reassigned HDDS-3261: - Assignee: Peter Orova > Fix TestOzoneManagerRestart.java > > > Key: HDDS-3261 > URL: https://issues.apache.org/jira/browse/HDDS-3261 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Shashikant Banerjee >Assignee: Peter Orova >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3259) Fix TestContainerReplicationEndToEnd.java
[ https://issues.apache.org/jira/browse/HDDS-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Orova resolved HDDS-3259. --- Resolution: Duplicate > Fix TestContainerReplicationEndToEnd.java > - > > Key: HDDS-3259 > URL: https://issues.apache.org/jira/browse/HDDS-3259 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Shashikant Banerjee >Assignee: Peter Orova >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3259) Fix TestContainerReplicationEndToEnd.java
[ https://issues.apache.org/jira/browse/HDDS-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170735#comment-17170735 ] Peter Orova commented on HDDS-3259: --- [~adoroszlai] that makes sense. Executing this repeatedly I did not manage to get a failure. > Fix TestContainerReplicationEndToEnd.java > - > > Key: HDDS-3259 > URL: https://issues.apache.org/jira/browse/HDDS-3259 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Shashikant Banerjee >Assignee: Peter Orova >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3259) Fix TestContainerReplicationEndToEnd.java
[ https://issues.apache.org/jira/browse/HDDS-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170732#comment-17170732 ] Attila Doroszlai commented on HDDS-3259: I think this is already fixed in duplicate issue. > Fix TestContainerReplicationEndToEnd.java > - > > Key: HDDS-3259 > URL: https://issues.apache.org/jira/browse/HDDS-3259 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Shashikant Banerjee >Assignee: Peter Orova >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3259) Fix TestContainerReplicationEndToEnd.java
[ https://issues.apache.org/jira/browse/HDDS-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Orova reassigned HDDS-3259: - Assignee: Peter Orova > Fix TestContainerReplicationEndToEnd.java > - > > Key: HDDS-3259 > URL: https://issues.apache.org/jira/browse/HDDS-3259 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Shashikant Banerjee >Assignee: Peter Orova >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3994) Write object when met exception can be slower than before
[
https://issues.apache.org/jira/browse/HDDS-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Jain resolved HDDS-3994.
---
Fix Version/s: 0.6.0
Resolution: Fixed
> Write object when met exception can be slower than before
> -
>
> Key: HDDS-3994
> URL: https://issues.apache.org/jira/browse/HDDS-3994
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.6.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.6.0
>
>
> After HDDS-3350 , the retry policy changed, and the client write performance
> getting lower than before.
>
> With HDDS-3350, I restore the method RatisHelper#createRetryPolicy to the
> previous commit, it works well.
>
> The previous is
>
> {code:java}
> static RetryPolicy createRetryPolicy(ConfigurationSource conf) {
> int maxRetryCount =
> conf.getInt(OzoneConfigKeys.DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_KEY,
> OzoneConfigKeys.
> DFS_RATIS_CLIENT_REQUEST_MAX_RETRIES_DEFAULT);
> long retryInterval = conf.getTimeDuration(OzoneConfigKeys.
> DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_KEY, OzoneConfigKeys.
> DFS_RATIS_CLIENT_REQUEST_RETRY_INTERVAL_DEFAULT
> .toIntExact(TimeUnit.MILLISECONDS), TimeUnit.MILLISECONDS);
> TimeDuration sleepDuration =
> TimeDuration.valueOf(retryInterval, TimeUnit.MILLISECONDS);
> RetryPolicy retryPolicy = RetryPolicies
> .retryUpToMaximumCountWithFixedSleep(maxRetryCount, sleepDuration);
> return retryPolicy;
> }
> {code}
> When I switch logLevel to TRACE level, i see the following log While using
> HDDS-3350
> 2020-07-21 12:56:11,822 [grpc-default-executor-5] TRACE impl.OrderedAsync:
> client-6F623ADF656D: Failed*
> RaftClientRequest:client-6F623ADF656D->207b98d9-ad64-45a8-940f-504b514feff5@group-83A28012848F,
> cid=2876, seq=1*, Watch(0), null
> java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.LeaderNotReadyException:
> 207b98d9-ad64-45a8-940f-504b514feff5@group-83A28012848F is in LEADER state
> but not ready yet.
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:358)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:264)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:283)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:269)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:436)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:658)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:643)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.ratis.protocol.LeaderNotReadyException:
> 207b98d9-ad64-45a8-940f-504b514feff5@group-83A28012848F is in LEADER state
> but not ready yet.
> at
> org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:281)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:274)
> ... 9 more
> 2020-07-21 12:56:11,822 [grpc-default-executor-5] DEBUG impl.OrderedAsync:
> schedule* attempt #1 with
[GitHub] [hadoop-ozone] lokeshj1703 commented on pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
lokeshj1703 commented on pull request #1231: URL: https://github.com/apache/hadoop-ozone/pull/1231#issuecomment-668488849 @maobaolong Thanks for the contribution! I have committed the PR to master branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 closed pull request #1231: HDDS-3994. Make retry policy can be set by configuration.
lokeshj1703 closed pull request #1231: URL: https://github.com/apache/hadoop-ozone/pull/1231 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3724) BackgroundPipelineCreator should not throw Pipeline creation failed exception due to insufficient nodes
[
https://issues.apache.org/jira/browse/HDDS-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170623#comment-17170623
]
Zheng Huang-Mu commented on HDDS-3724:
--
It's seems to be fixed by [~xyao] in HDDS-4027?
> BackgroundPipelineCreator should not throw Pipeline creation failed exception
> due to insufficient nodes
> ---
>
> Key: HDDS-3724
> URL: https://issues.apache.org/jira/browse/HDDS-3724
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Priority: Major
>
> BackgroundPipelineCreator runs a scheduler to keep creating pipelines if
> possible. But in a healthy cluster where all the datanodes are part of
> pipelines and there are no more nodes available for creating new pipelines,
> the BackgroundPipelineCreator should not log an error message. In this
> scenario, it is not an error that a pipeline couldn't be created.
> SCM logs were flooded with "Failed to create pipeline" error messages because
> of this:
> {code:java}
> 2020-06-04 11:03:24,229 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
> due to no sufficient healthy datanodes. Required 3. Found 1.
> 2020-06-04 11:05:24,231 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
> factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
> 2020-06-04 11:05:24,231 [RatisPipelineUtilsThread] WARN
> org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline
> creation failed due to no sufficient healthy datanodes. Required 3. Found 1.
> 2020-06-04 11:05:24,231 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
> due to no sufficient healthy datanodes. Required 3. Found 1.
> 2020-06-04 11:05:40,705 [ReplicationMonitor] INFO
> org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor
> Thread took 0 milliseconds for processing 7 containers.
> 2020-06-04 11:07:24,234 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
> factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
> 2020-06-04 11:07:24,234 [RatisPipelineUtilsThread] WARN
> org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline
> creation failed due to no sufficient healthy datanodes. Required 3. Found 1.
> 2020-06-04 11:07:24,234 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
> due to no sufficient healthy datanodes. Required 3. Found 1.
> 2020-06-04 11:09:24,242 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
> factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
> 2020-06-04 11:09:24,242 [RatisPipelineUtilsThread] WARN
> org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline
> creation failed due to no sufficient healthy datanodes. Required 3. Found 1.
> 2020-06-04 11:09:24,242 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
> due to no sufficient healthy datanodes. Required 3. Found 1.
> 2020-06-04 11:10:40,700 [ReplicationMonitor] INFO
> org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor
> Thread took 0 milliseconds for processing 7 containers.
> 2020-06-04 11:11:24,234 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor ONE. Exception: Cannot create pipeline of
> factor 1 using 0 nodes. Used 1 nodes. Healthy nodes 1
> 2020-06-04 11:11:24,235 [RatisPipelineUtilsThread] WARN
> org.apache.hadoop.hdds.scm.pipeline.PipelinePlacementPolicy: Pipeline
> creation failed due to no sufficient healthy datanodes. Required 3. Found 1.
> 2020-06-04 11:11:24,235 [RatisPipelineUtilsThread] ERROR
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Failed to create
> pipeline of type RATIS and factor THREE. Exception: Pipeline creation failed
> due to no sufficient healthy datanodes. Required 3. Found 1.
> 2020-06-04 11:13:24,240 [RatisPipelineUtilsThread] ERROR
[GitHub] [hadoop-ozone] timmylicheng closed pull request #1191: HDDS-3837 Add isLeader check in SCMHAManager.
timmylicheng closed pull request #1191: URL: https://github.com/apache/hadoop-ozone/pull/1191 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
