[jira] [Updated] (HDDS-3285) MiniOzoneChaosCluster exits because of deadline exceeding
[
https://issues.apache.org/jira/browse/HDDS-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDDS-3285:
Priority: Blocker (was: Major)
> MiniOzoneChaosCluster exits because of deadline exceeding
> -
>
> Key: HDDS-3285
> URL: https://issues.apache.org/jira/browse/HDDS-3285
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Blocker
> Labels: MiniOzoneChaosCluster
> Attachments: complete.log.gz
>
>
> 2020-03-26 21:26:48,869 [pool-326-thread-2] INFO util.ExitUtil
> (ExitUtil.java:terminate(210)) - Exiting with status 1: java.io.IOException:
> java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.
> grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall started after
> deadline exceeded: -4.330590725s from now
> {code}
> 2020-03-26 21:26:48,866 [pool-326-thread-2] ERROR
> loadgenerators.LoadExecutors (LoadExecutors.java:load(64)) - FileSystem
> LOADGEN: null Exiting due to exception
> java.io.IOException: java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
> DEADLINE_EXCEEDED: ClientCall started after deadline exceeded: -4.330590725s
> from now
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:359)
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:281)
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:259)
> at
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:119)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:199)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:133)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:254)
> at
> org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:197)
> at
> org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:63)
> at java.io.DataInputStream.read(DataInputStream.java:100)
> at
> org.apache.hadoop.ozone.utils.LoadBucket$ReadOp.doPostOp(LoadBucket.java:205)
> at
> org.apache.hadoop.ozone.utils.LoadBucket$Op.execute(LoadBucket.java:121)
> at
> org.apache.hadoop.ozone.utils.LoadBucket$ReadOp.execute(LoadBucket.java:180)
> at
> org.apache.hadoop.ozone.utils.LoadBucket.readKey(LoadBucket.java:82)
> at
> org.apache.hadoop.ozone.loadgenerators.FilesystemLoadGenerator.generateLoad(FilesystemLoadGenerator.java:54)
> at
> org.apache.hadoop.ozone.loadgenerators.LoadExecutors.load(LoadExecutors.java:62)
> at
> org.apache.hadoop.ozone.loadgenerators.LoadExecutors.lambda$startLoad$0(LoadExecutors.java:78)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
> DEADLINE_EXCEEDED: ClientCall started after deadline exceeded: -4.330590725s
> from now
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:336)
> ... 20 more
> Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
> DEADLINE_EXCEEDED: ClientCall started after deadline exceeded: -4.330590725s
> from now
> at
> org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:533)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
> at
> org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
>at
> org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsMo
[GitHub] [hadoop-ozone] timmylicheng commented on issue #668: HDDS-3139 Pipeline placement should max out pipeline usage
timmylicheng commented on issue #668: HDDS-3139 Pipeline placement should max out pipeline usage URL: https://github.com/apache/hadoop-ozone/pull/668#issuecomment-607065266 > Hi @timmylicheng this patch is giving some conflicts now, probably as we merged the other pipeline related change. Could you rebase against master and push it again please? @sodonnel I rebase the code with minor conflicts in test class, but the test won't pass. I took a close look and made some change. But I realize the issue that I mention in the last comment about how to leverage with chooseNodeFromTopology. Wanna learn your thoughts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup
bharatviswa504 commented on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup URL: https://github.com/apache/hadoop-ozone/pull/740#issuecomment-607014247 > > @runzhiwang Thanks for the PR and improvement. > > I have few comments. > > I have one general question, now in native authorizer, the hostname is not used during checkAcls, in the future if we use, allow requests from this host like settings, then with this approach, someone can set the hostName, even though the request is not from that host. This can happen with this code change, I am not sure we will support this kind of scenario or not. Just flagging this here. > > @bharatviswa504 Thanks for your review. Very reasonable question. And I also doubt whether we need hostName in authorizer, because `getHostName `from IP is a block call, maybe block very long time. If we do not need hostName in authorizer, we can delete `getHostName `and also no need to `setHostName `in client. HostName is added based on the comments from @eyang during the review of HDDS-1600. Have a look into it. From reading the current code, in NativeAuthorizer, we have not used hostName during ACL validation. MIght be some external providers like ranger might need this, not sure ranger is currently using it or not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang edited a comment on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup
runzhiwang edited a comment on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup URL: https://github.com/apache/hadoop-ozone/pull/740#issuecomment-607011449 > @runzhiwang Thanks for the PR and improvement. > > I have few comments. > > I have one general question, now in native authorizer, the hostname is not used during checkAcls, in the future if we use, allow requests from this host like settings, then with this approach, someone can set the hostName, even though the request is not from that host. This can happen with this code change, I am not sure we will support this kind of scenario or not. Just flagging this here. @bharatviswa504 Thanks for your review. Very reasonable question. And I also doubt whether we need hostName in authorizer, because `getHostName `from IP is a block call, maybe block a very long time. If we do not need hostName in authorizer, we can delete `getHostName `and also no need to `setHostName `in client. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang edited a comment on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup
runzhiwang edited a comment on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup URL: https://github.com/apache/hadoop-ozone/pull/740#issuecomment-607011449 > @runzhiwang Thanks for the PR and improvement. > > I have few comments. > > I have one general question, now in native authorizer, the hostname is not used during checkAcls, in the future if we use, allow requests from this host like settings, then with this approach, someone can set the hostName, even though the request is not from that host. This can happen with this code change, I am not sure we will support this kind of scenario or not. Just flagging this here. @bharatviswa504 Thanks for your review. Very reasonable question. And I also doubt whether we need hostName in authorizer, because `getHostName `from IP is a block call, maybe block very long time. If we do not need hostName in authorizer, we can delete `getHostName `and also no need to `setHostName `in client. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup
runzhiwang commented on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup URL: https://github.com/apache/hadoop-ozone/pull/740#issuecomment-607011449 > @runzhiwang Thanks for the PR and improvement. > > I have few comments. > > I have one general question, now in native authorizer, the hostname is not used during checkAcls, in the future if we use, allow requests from this host like settings, then with this approach, someone can set the hostName, even though the request is not from that host. This can happen with this code change, I am not sure we will support this kind of scenario or not. Just flagging this here. @bharatviswa504 Thanks for your review. Very reasonable question. And I also doubt whether we need hostName in authorizer, because getHostName() from IP is a block call, maybe block a very long time, it's dangerous. If we do not need hostName in authorizer, we can delete `getHostName `and also no need to `setHostName `in client. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on issue #741: HDDS-3308. Update Ratis snapshot
mukul1987 commented on issue #741: HDDS-3308. Update Ratis snapshot URL: https://github.com/apache/hadoop-ozone/pull/741#issuecomment-607008034 The test failure seems related as a ratis test is failing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3001) NFS support for Ozone
[ https://issues.apache.org/jira/browse/HDDS-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072351#comment-17072351 ] Mukul Kumar Singh commented on HDDS-3001: - I have created HDDS-3001 branch for this work. > NFS support for Ozone > - > > Key: HDDS-3001 > URL: https://issues.apache.org/jira/browse/HDDS-3001 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Filesystem >Affects Versions: 0.5.0 >Reporter: Prashant Pogde >Assignee: Prashant Pogde >Priority: Major > Attachments: NFS Support for Ozone.pdf > > > Provide NFS support for Ozone -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3289) Add a freon generator to create nested directories
[ https://issues.apache.org/jira/browse/HDDS-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-3289. - Fix Version/s: 0.6.0 Target Version/s: 0.6.0 Resolution: Fixed Merged this to master. Thanks [~aryangupta1998] for the improvement. Thanks [~adoroszlai] & [~rakeshr] for the reviews. > Add a freon generator to create nested directories > -- > > Key: HDDS-3289 > URL: https://issues.apache.org/jira/browse/HDDS-3289 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Tools >Reporter: Aryan Gupta >Assignee: Aryan Gupta >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > This Jira proposes to add a functionality to freon to create nested > directories. Also, multiple child directories can be created inside the leaf > directory and also multiple top level directories can be created. > This functionality will help to determine the scalability of ozone by > creating nested directories(may be in millions) and also it will allow to > control nested directories properties, just by passing appropriate > parameters. This all can be done just by executing a single command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3289) Add a freon generator to create nested directories
[ https://issues.apache.org/jira/browse/HDDS-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-3289: Issue Type: Improvement (was: Bug) > Add a freon generator to create nested directories > -- > > Key: HDDS-3289 > URL: https://issues.apache.org/jira/browse/HDDS-3289 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Tools >Reporter: Aryan Gupta >Assignee: Aryan Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > This Jira proposes to add a functionality to freon to create nested > directories. Also, multiple child directories can be created inside the leaf > directory and also multiple top level directories can be created. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3289) Add a freon generator to create nested directories
[ https://issues.apache.org/jira/browse/HDDS-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-3289: Description: This Jira proposes to add a functionality to freon to create nested directories. Also, multiple child directories can be created inside the leaf directory and also multiple top level directories can be created. This functionality will help to determine the scalability of ozone by creating nested directories(may be in millions) and also it will allow to control nested directories properties, just by passing appropriate parameters. This all can be done just by executing a single command. was:This Jira proposes to add a functionality to freon to create nested directories. Also, multiple child directories can be created inside the leaf directory and also multiple top level directories can be created. > Add a freon generator to create nested directories > -- > > Key: HDDS-3289 > URL: https://issues.apache.org/jira/browse/HDDS-3289 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Tools >Reporter: Aryan Gupta >Assignee: Aryan Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > This Jira proposes to add a functionality to freon to create nested > directories. Also, multiple child directories can be created inside the leaf > directory and also multiple top level directories can be created. > This functionality will help to determine the scalability of ozone by > creating nested directories(may be in millions) and also it will allow to > control nested directories properties, just by passing appropriate > parameters. This all can be done just by executing a single command. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #730: HDDS-3289. Add a freon generator to create nested directories.
dineshchitlangia merged pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup
bharatviswa504 commented on issue #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup URL: https://github.com/apache/hadoop-ozone/pull/740#issuecomment-606946772 And also test failure looks related to the code change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup
bharatviswa504 commented on a change in pull request #740: HDDS-3307. Improve
write efficiency by avoiding reverse DNS lookup
URL: https://github.com/apache/hadoop-ozone/pull/740#discussion_r401275915
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##
@@ -127,7 +127,14 @@ public OMRequest getOmRequest() {
}
if (remoteAddress != null) {
- userInfo.setHostName(remoteAddress.getHostName());
+ OzoneManagerProtocolProtos.UserInfo userInfoInProtocol = getOmRequest()
+ .toBuilder().getUserInfo();
+ if (userInfoInProtocol.getHostName().isEmpty()) {
Review comment:
If userInfoInProtocol.getHostName(), we need to set it as before right?
Like userInfo.setHostName(remoteAddress.getHostName());
As when someone uses RpcClient, this if check will not be true. So if
HostName is empty from UserInfo, I think we still need to set hostname as
before.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup
bharatviswa504 commented on a change in pull request #740: HDDS-3307. Improve
write efficiency by avoiding reverse DNS lookup
URL: https://github.com/apache/hadoop-ozone/pull/740#discussion_r401276822
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java
##
@@ -127,7 +127,14 @@ public OMRequest getOmRequest() {
}
if (remoteAddress != null) {
- userInfo.setHostName(remoteAddress.getHostName());
+ OzoneManagerProtocolProtos.UserInfo userInfoInProtocol = getOmRequest()
Review comment:
We don't need toBuilder here, we can get UserInfo by calling getUserInfo()
Like `getOmRequest().getUserInfo()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] fapifta commented on issue #719: HDDS-3270. Allow safemode listeners to be notified when some precheck rules pass
fapifta commented on issue #719: HDDS-3270. Allow safemode listeners to be notified when some precheck rules pass URL: https://github.com/apache/hadoop-ozone/pull/719#issuecomment-606929024 Hi @sodonnel thank you for addressing the concerns, and thank you for working on the changes. The patch in its current form looks good to me, +1 (non-binding) to commit it ;) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #696: HDDS-3056. Allow users to list volumes they have access to, and optionally allow all users to list all volumes
smengcl commented on a change in pull request #696: HDDS-3056. Allow users to
list volumes they have access to, and optionally allow all users to list all
volumes
URL: https://github.com/apache/hadoop-ozone/pull/696#discussion_r401245510
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -1819,7 +1823,30 @@ public void deleteVolume(String volume) throws
IOException {
auditMap.put(OzoneConsts.USERNAME, userName);
try {
metrics.incNumVolumeLists();
- return volumeManager.listVolumes(userName, prefix, prevKey, maxKeys);
+ if (isAclEnabled) {
+if (LOG.isDebugEnabled()) {
+ LOG.debug("ACL is enabled. Listing volumes accessible by user. "
+ + "Principal: {}, keytab: {}",
+ configuration.get(OZONE_OM_KERBEROS_PRINCIPAL_KEY),
+ configuration.get(OZONE_OM_KERBEROS_KEYTAB_FILE_KEY));
+}
+// List all volumes first
+List listOfAllVolumes = volumeManager.listVolumes(
+null, prefix, prevKey, maxKeys);
+// Filter all volumes by ACL LIST permission of UGI
+return listOfAllVolumes.stream().filter(v -> v.getAclMap()
+.hasAccess(IAccessAuthorizer.ACLType.LIST, remoteUserUgi))
+.collect(Collectors.toList());
+ } else {
+if (LOG.isDebugEnabled()) {
Review comment:
Remove log
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #696: HDDS-3056. Allow users to list volumes they have access to, and optionally allow all users to list all volumes
smengcl commented on a change in pull request #696: HDDS-3056. Allow users to
list volumes they have access to, and optionally allow all users to list all
volumes
URL: https://github.com/apache/hadoop-ozone/pull/696#discussion_r401245463
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -1819,7 +1823,30 @@ public void deleteVolume(String volume) throws
IOException {
auditMap.put(OzoneConsts.USERNAME, userName);
try {
metrics.incNumVolumeLists();
- return volumeManager.listVolumes(userName, prefix, prevKey, maxKeys);
+ if (isAclEnabled) {
+if (LOG.isDebugEnabled()) {
+ LOG.debug("ACL is enabled. Listing volumes accessible by user. "
+ + "Principal: {}, keytab: {}",
+ configuration.get(OZONE_OM_KERBEROS_PRINCIPAL_KEY),
+ configuration.get(OZONE_OM_KERBEROS_KEYTAB_FILE_KEY));
+}
+// List all volumes first
+List listOfAllVolumes = volumeManager.listVolumes(
+null, prefix, prevKey, maxKeys);
+// Filter all volumes by ACL LIST permission of UGI
+return listOfAllVolumes.stream().filter(v -> v.getAclMap()
Review comment:
Use `accessAuthorizer.checkAccess`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #696: HDDS-3056. Allow users to list volumes they have access to, and optionally allow all users to list all volumes
smengcl commented on a change in pull request #696: HDDS-3056. Allow users to
list volumes they have access to, and optionally allow all users to list all
volumes
URL: https://github.com/apache/hadoop-ozone/pull/696#discussion_r401242906
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -1819,7 +1823,30 @@ public void deleteVolume(String volume) throws
IOException {
auditMap.put(OzoneConsts.USERNAME, userName);
try {
metrics.incNumVolumeLists();
- return volumeManager.listVolumes(userName, prefix, prevKey, maxKeys);
+ if (isAclEnabled) {
+if (LOG.isDebugEnabled()) {
Review comment:
Remove log
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2592) Add Datanode command to allow the datanode to persist its admin state
[ https://issues.apache.org/jira/browse/HDDS-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDDS-2592: Resolution: Fixed Status: Resolved (was: Patch Available) > Add Datanode command to allow the datanode to persist its admin state > -- > > Key: HDDS-2592 > URL: https://issues.apache.org/jira/browse/HDDS-2592 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode, SCM >Affects Versions: 0.5.0 >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When the operational state of a datanode changes, an async command should be > triggered to persist the new state on the datanodes. For maintenance mode, > the datanode should also store the maintenance end time. The datanode will > then report the new state (and optional maintenance end time) back via its > heartbeat. > The purpose of the DN persisting this information and heartbeating it back to > SCM is to allow the operation state to be recovered after a SCM reboot, as > SCM does not persist any of this information. It also allows "Recon" to learn > the datanode states. > If SCM is restarted, then it will forget all knowledge of the datanodes. When > they register, their operational state will be reported and SCM can set it > correctly. > Outside of registration (ie during normal heartbeats), the SCM state is the > source of truth for the operational state and if the DN heartbeat reports a > state that is not the same as SCM, SCM should issue another command to the > datanode to set its state to the SCM value. There is a chance the state miss > match is due to an unprocessed command triggered by the SCM state change, but > the worst case is an extra command sent to the datanode. This is a very > lightweight command, so that is not an issue. > One open question is whether to persist intermediate states on the DN. Ie for > decommissioning, the DN will first persist "Decommissioning" and then > transition to "Decommissioned" when SCM is satisfied all containers are > replicated. It would be possible to persist both these states in turn on the > datanode quite easily in turn. Or, we set the end state (Decommissioned) on > the datanode and allow SCM to get the node to that state. For the latter, if > SCM is restarted, then the DN will report "Decommissioned" on registration, > but SCM will set its internal state to Decommissioning and then ensure all > containers are replicated before transitioning the node to Decommissioned. > This seems like a safer approach, but there are advantages of tracking the > intermediate states on the DNs too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #696: HDDS-3056. Allow users to list volumes they have access to, and optionally allow all users to list all volumes
smengcl commented on a change in pull request #696: HDDS-3056. Allow users to
list volumes they have access to, and optionally allow all users to list all
volumes
URL: https://github.com/apache/hadoop-ozone/pull/696#discussion_r401240918
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java
##
@@ -76,6 +76,10 @@ private OMConfigKeys() {
"ozone.om.db.cache.size.mb";
public static final int OZONE_OM_DB_CACHE_SIZE_DEFAULT = 128;
+ public static final String OZONE_OM_USER_VOLUME_LISTALL_ALLOWED =
Review comment:
Remove `_USER`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel merged pull request #521: HDDS-2592 Add Datanode command to allow the datanode to persist its admin state
sodonnel merged pull request #521: HDDS-2592 Add Datanode command to allow the datanode to persist its admin state URL: https://github.com/apache/hadoop-ozone/pull/521 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 removed a comment on issue #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 removed a comment on issue #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730#issuecomment-606882774 > @aryangupta1998 Thanks for working on this. > Aside from the changes requested in the review, could you also update the PR template and JIRA to explain why we need this change, what benefit does this bring or what problem does it solve. This will help the reviewers to understand your proposal and merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401225491 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java ## @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.ozone.freon; + +import java.net.URI; +import java.util.concurrent.Callable; + +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdds.cli.HddsVersionProvider; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; + +import org.apache.commons.lang3.RandomStringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import picocli.CommandLine.Command; +import picocli.CommandLine.Option; + +/** + * Directory generator tool test om performance. Review comment: Changed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 opened a new pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 opened a new pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730 ## What changes were proposed in this pull request? This Jira proposes to add a functionality to freon to create nested directories. Also, multiple child directories can be created inside the leaf directory and also multiple top level directories can be created. This functionality will help to determine the scalability of ozone by creating nested directories(may be in millions) and also it will allow to control nested directories properties, just by passing appropriate parameters. This all can be done just by executing a single command. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3289 ## How was this patch tested? Tested manually by running Freon Directory Generator. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 closed pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 closed pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on issue #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on issue #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730#issuecomment-606882774 > @aryangupta1998 Thanks for working on this. > Aside from the changes requested in the review, could you also update the PR template and JIRA to explain why we need this change, what benefit does this bring or what problem does it solve. This will help the reviewers to understand your proposal and merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel commented on a change in pull request #719: HDDS-3270. Allow safemode listeners to be notified when some precheck rules pass
sodonnel commented on a change in pull request #719: HDDS-3270. Allow safemode
listeners to be notified when some precheck rules pass
URL: https://github.com/apache/hadoop-ozone/pull/719#discussion_r401221058
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -641,13 +663,26 @@ public boolean getSafeModeStatus() {
}
@Override
- public void handleSafeModeTransition(
+ public synchronized void handleSafeModeTransition(
SCMSafeModeManager.SafeModeStatus status) {
-this.isInSafeMode.set(status.getSafeModeStatus());
-if (!status.getSafeModeStatus()) {
- // TODO: #CLUTIL if we reenter safe mode the fixed interval pipeline
- // creation job needs to stop
+// TODO: #CLUTIL - handle safemode getting re-enabled
+boolean currentAllowPipelines =
+allowPipelineCreation.getAndSet(status.isPreCheckComplete());
+boolean currentlyInSafeMode =
+isInSafeMode.getAndSet(status.isInSafeMode());
+boolean triggerPipelines = false;
+
+// Trigger pipeline creation only if the preCheck status has changed to
+// complete.
+if (allowPipelineCreation.get() && !currentAllowPipelines) {
+ triggerPipelines = true;
+}
+// Start the pipeline creation thread only when safemode switches off
+if (!getSafeModeStatus() && currentlyInSafeMode) {
startPipelineCreator();
+ triggerPipelines = true;
+}
+if (triggerPipelines) {
Review comment:
Good observation - you are correct. I have simplified this as you suggested.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401206949 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java ## @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.ozone.freon; + +import java.net.URI; +import java.util.concurrent.Callable; + +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdds.cli.HddsVersionProvider; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; + +import org.apache.commons.lang3.RandomStringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import picocli.CommandLine.Command; +import picocli.CommandLine.Option; + +/** + * Directory generator tool test om performance. + */ +@Command(name = "ddsg", Review comment: As far as I know Freon commands have cryptic names therefore I have given "ddsg", also I have given readable name "dfs-directory-generator". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401204828
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
+ description = "Hadoop FS directory system path",
+ defaultValue = "o3fs://bucket2.vol2")
+ private String rootPath;
+
+
+ @Option(names = {"-d", "--depth"},
+ description = "Number of directories to be generated recursively",
+ defaultValue = "5")
+ private int depth;
+
+ @Option(names = {"--span"},
+ description =
+ "Number of child directories to be created in leaf directory.",
+ defaultValue = "10")
+ private int span;
+
+ @Option(names = {"--nameLen"},
+ description =
+ "Length of the random name of directory you want to create.",
+ defaultValue = "10")
+ private int n;
Review comment:
Changed the variable 'n' to 'length'.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401204575
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
+ description = "Hadoop FS directory system path",
+ defaultValue = "o3fs://bucket2.vol2")
+ private String rootPath;
+
+
+ @Option(names = {"-d", "--depth"},
+ description = "Number of directories to be generated recursively",
+ defaultValue = "5")
+ private int depth;
+
+ @Option(names = {"--span"},
+ description =
+ "Number of child directories to be created in leaf directory.",
+ defaultValue = "10")
+ private int span;
+
+ @Option(names = {"--nameLen"},
Review comment:
Added short name option "-l" for length.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401204342
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
+ description = "Hadoop FS directory system path",
+ defaultValue = "o3fs://bucket2.vol2")
+ private String rootPath;
+
+
+ @Option(names = {"-d", "--depth"},
+ description = "Number of directories to be generated recursively",
+ defaultValue = "5")
+ private int depth;
+
+ @Option(names = {"--span"},
Review comment:
Added short name option "-s" for span.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401203996
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
+ description = "Hadoop FS directory system path",
+ defaultValue = "o3fs://bucket2.vol2")
Review comment:
If volume or bucket does not exist then command will not execute
successfully and it will show number of successful executions=0.
Also default value just assumes a bucket and volume, you can pass your own
bucket and volume(which already exists) with "-r" or "--rpath" parameter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401201973
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
Review comment:
As -p is already taken for prefix, I changed --dpath to --rpath(root path)
and added short name option -r
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3309) Add timeouts to freon tests
Siyao Meng created HDDS-3309: Summary: Add timeouts to freon tests Key: HDDS-3309 URL: https://issues.apache.org/jira/browse/HDDS-3309 Project: Hadoop Distributed Data Store Issue Type: Test Components: test Reporter: Siyao Meng Assignee: Siyao Meng e.g. TestRandomKeyGenerator -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3217) Datanode startup is slow due to iterating container DB 2-3 times
[
https://issues.apache.org/jira/browse/HDDS-3217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-3217:
-
Labels: billiontest pull-request-available (was: billiontest)
> Datanode startup is slow due to iterating container DB 2-3 times
>
>
> Key: HDDS-3217
> URL: https://issues.apache.org/jira/browse/HDDS-3217
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Critical
> Labels: billiontest, pull-request-available
>
> During Datanode startup, for each container we iterate 2 times entire DB
> 1. For Setting block length
> 2. For finding delete Key count.
> And for open containers, we do step 1 again.
> *Code Snippet:*
> *ContainerReader.java:*
> *For setting Bytes Used:*
> {code:java}
> List> liveKeys = metadata.getStore()
> .getRangeKVs(null, Integer.MAX_VALUE,
> MetadataKeyFilters.getNormalKeyFilter());
> bytesUsed = liveKeys.parallelStream().mapToLong(e-> {
> BlockData blockData;
> try {
> blockData = BlockUtils.getBlockData(e.getValue());
> return blockData.getSize();
> } catch (IOException ex) {
> return 0L;
> }
> }).sum();
> kvContainerData.setBytesUsed(bytesUsed);
> {code}
> *For setting pending deleted Key count*
> {code:java}
> MetadataKeyFilters.KeyPrefixFilter filter =
> new MetadataKeyFilters.KeyPrefixFilter()
> .addFilter(OzoneConsts.DELETING_KEY_PREFIX);
> int numPendingDeletionBlocks =
> containerDB.getStore().getSequentialRangeKVs(null,
> Integer.MAX_VALUE, filter)
> .size();
> kvContainerData.incrPendingDeletionBlocks(numPendingDeletionBlocks);
> {code}
> *For open Containers*
> {code:java}
> if (kvContainer.getContainerState()
> == ContainerProtos.ContainerDataProto.State.OPEN) {
> // commitSpace for Open Containers relies on usedBytes
> initializeUsedBytes(kvContainer);
> }
> {code}
> *Jstack of DN during startup*
> {code:java}
> "Thread-8" #34 prio=5 os_prio=0 tid=0x7f5df507 nid=0x8ee runnable
> [0x7f4d840f3000]
>java.lang.Thread.State: RUNNABLE
> at org.rocksdb.RocksIterator.next0(Native Method)
> at
> org.rocksdb.AbstractRocksIterator.next(AbstractRocksIterator.java:70)
> at
> org.apache.hadoop.hdds.utils.RocksDBStore.getRangeKVs(RocksDBStore.java:195)
> at
> org.apache.hadoop.hdds.utils.RocksDBStore.getRangeKVs(RocksDBStore.java:155)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:158)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyAndFixupContainerData(ContainerReader.java:191)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.verifyContainerFile(ContainerReader.java:168)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.readVolume(ContainerReader.java:146)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerReader.run(ContainerReader.java:101)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #742: HDDS-3217. Datanode startup is slow due to iterating container DB 2-3 times.
bharatviswa504 opened a new pull request #742: HDDS-3217. Datanode startup is slow due to iterating container DB 2-3 times. URL: https://github.com/apache/hadoop-ozone/pull/742 ## What changes were proposed in this pull request? Reduce the time spent during DN restart. Commit the required info which is needed during DN restart commit it to container DB. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3217 ## How was this patch tested? Updated Block Deletion test, will check if any more tests can be added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3308) Update Ratis snapshot
[
https://issues.apache.org/jira/browse/HDDS-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-3308:
---
Status: Patch Available (was: In Progress)
> Update Ratis snapshot
> -
>
> Key: HDDS-3308
> URL: https://issues.apache.org/jira/browse/HDDS-3308
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: build
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Update Ratis snapshot version to {{7c5b30d}}, which includes RATIS-816,
> required for HDDS-3023.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3308) Update Ratis snapshot
[
https://issues.apache.org/jira/browse/HDDS-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-3308:
-
Labels: pull-request-available (was: )
> Update Ratis snapshot
> -
>
> Key: HDDS-3308
> URL: https://issues.apache.org/jira/browse/HDDS-3308
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: build
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
>
> Update Ratis snapshot version to {{7c5b30d}}, which includes RATIS-816,
> required for HDDS-3023.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #741: HDDS-3308. Update Ratis snapshot
adoroszlai opened a new pull request #741: HDDS-3308. Update Ratis snapshot URL: https://github.com/apache/hadoop-ozone/pull/741 ## What changes were proposed in this pull request? Update Ratis snapshot version to `7c5b30d`, which includes [RATIS-816](https://issues.apache.org/jira/browse/RATIS-816), required for [HDDS-3023](https://issues.apache.org/jira/browse/HDDS-3023). https://issues.apache.org/jira/browse/HDDS-3308 ## How was this patch tested? CI: https://github.com/adoroszlai/hadoop-ozone/runs/549579172 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
adoroszlai commented on a change in pull request #730: HDDS-3289. Add a freon
generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401106743
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
Review comment:
`-p` is already taken for `--prefix`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
adoroszlai commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401106180 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java ## @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.ozone.freon; + +import java.net.URI; +import java.util.concurrent.Callable; + +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdds.cli.HddsVersionProvider; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; + +import org.apache.commons.lang3.RandomStringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import picocli.CommandLine.Command; +import picocli.CommandLine.Option; + +/** + * Directory generator tool test om performance. + */ +@Command(name = "ddsg", Review comment: Freon commands have cryptic names (`ockg`, `dfsv`, etc.), but they also have more readable aliases. This one also does: [`dfs-directory-generator`](https://github.com/apache/hadoop-ozone/blob/9861e98c630a1843132c590f1fffa0e20540675a/hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java#L37). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3308) Update Ratis snapshot
Attila Doroszlai created HDDS-3308:
--
Summary: Update Ratis snapshot
Key: HDDS-3308
URL: https://issues.apache.org/jira/browse/HDDS-3308
Project: Hadoop Distributed Data Store
Issue Type: Task
Components: build
Reporter: Attila Doroszlai
Assignee: Attila Doroszlai
Update Ratis snapshot version to {{7c5b30d}}, which includes RATIS-816,
required for HDDS-3023.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401095527
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
Review comment:
Add short name option `-p` for path
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401096466
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
+ description = "Hadoop FS directory system path",
+ defaultValue = "o3fs://bucket2.vol2")
+ private String rootPath;
+
+
+ @Option(names = {"-d", "--depth"},
+ description = "Number of directories to be generated recursively",
+ defaultValue = "5")
+ private int depth;
+
+ @Option(names = {"--span"},
Review comment:
Add short name option -s for span
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401096255
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
+ description = "Hadoop FS directory system path",
+ defaultValue = "o3fs://bucket2.vol2")
Review comment:
Default value assumes the bucket name and volume name. What happens if this
volume or bucket does not exist?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401097666
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
+ description = "Hadoop FS directory system path",
+ defaultValue = "o3fs://bucket2.vol2")
+ private String rootPath;
+
+
+ @Option(names = {"-d", "--depth"},
+ description = "Number of directories to be generated recursively",
+ defaultValue = "5")
+ private int depth;
+
+ @Option(names = {"--span"},
+ description =
+ "Number of child directories to be created in leaf directory.",
+ defaultValue = "10")
+ private int span;
+
+ @Option(names = {"--nameLen"},
+ description =
+ "Length of the random name of directory you want to create.",
+ defaultValue = "10")
+ private int n;
Review comment:
Variable `n` is not understandable. Replace with more appropriate name like
`length`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401097065
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopNestedDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopNestedDirGenerator.class);
+
+ @Option(names = {"--dpath"},
+ description = "Hadoop FS directory system path",
+ defaultValue = "o3fs://bucket2.vol2")
+ private String rootPath;
+
+
+ @Option(names = {"-d", "--depth"},
+ description = "Number of directories to be generated recursively",
+ defaultValue = "5")
+ private int depth;
+
+ @Option(names = {"--span"},
+ description =
+ "Number of child directories to be created in leaf directory.",
+ defaultValue = "10")
+ private int span;
+
+ @Option(names = {"--nameLen"},
Review comment:
Add short name option `-l` for length.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401094279 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java ## @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.ozone.freon; + +import java.net.URI; +import java.util.concurrent.Callable; + +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdds.cli.HddsVersionProvider; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; + +import org.apache.commons.lang3.RandomStringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import picocli.CommandLine.Command; +import picocli.CommandLine.Option; + +/** + * Directory generator tool test om performance. Review comment: NIT: tool `to` test `OM` performance This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3001) NFS support for Ozone
[ https://issues.apache.org/jira/browse/HDDS-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072013#comment-17072013 ] Prashant Pogde commented on HDDS-3001: -- [~elek] We plan to make random writes work similar to HDFS. > NFS support for Ozone > - > > Key: HDDS-3001 > URL: https://issues.apache.org/jira/browse/HDDS-3001 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Filesystem >Affects Versions: 0.5.0 >Reporter: Prashant Pogde >Assignee: Prashant Pogde >Priority: Major > Attachments: NFS Support for Ozone.pdf > > > Provide NFS support for Ozone -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
dineshchitlangia commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r401095167 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopNestedDirGenerator.java ## @@ -0,0 +1,113 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.ozone.freon; + +import java.net.URI; +import java.util.concurrent.Callable; + +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdds.cli.HddsVersionProvider; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; + +import org.apache.commons.lang3.RandomStringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import picocli.CommandLine.Command; +import picocli.CommandLine.Option; + +/** + * Directory generator tool test om performance. + */ +@Command(name = "ddsg", Review comment: Replace command name with something less cryptic, example dirgen, gendir as we have similar commands like genconf etc. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2339) Add OzoneManager to MiniOzoneChaosCluster
[ https://issues.apache.org/jira/browse/HDDS-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh resolved HDDS-2339. - Fix Version/s: 0.6.0 Resolution: Fixed Thanks for the contribution [~hanishakoneru] and [~nanda] for the review. I have merged this. > Add OzoneManager to MiniOzoneChaosCluster > - > > Key: HDDS-2339 > URL: https://issues.apache.org/jira/browse/HDDS-2339 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: om >Reporter: Mukul Kumar Singh >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > This jira proposes to add OzoneManager to MiniOzoneChaosCluster with OzoneHA > implementation done. This will help in discovering bugs in Ozone Manager HA -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 merged pull request #643: HDDS-2339. Add OzoneManager to MiniOzoneChaosCluster
mukul1987 merged pull request #643: HDDS-2339. Add OzoneManager to MiniOzoneChaosCluster URL: https://github.com/apache/hadoop-ozone/pull/643 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3156) update allocateContainer to remove additional createPipeline step.
[ https://issues.apache.org/jira/browse/HDDS-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar resolved HDDS-3156. --- Fix Version/s: 0.6.0 Resolution: Fixed > update allocateContainer to remove additional createPipeline step. > -- > > Key: HDDS-3156 > URL: https://issues.apache.org/jira/browse/HDDS-3156 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Li Cheng >Assignee: Li Cheng >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > AllocateContainer now tries allocatePipelines. But with multi-raft, it should > not worry about whether there are available pipelines. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 merged pull request #663: HDDS-3156 update allocateContainer to remove additional createPipeline step.
nandakumar131 merged pull request #663: HDDS-3156 update allocateContainer to remove additional createPipeline step. URL: https://github.com/apache/hadoop-ozone/pull/663 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on issue #663: HDDS-3156 update allocateContainer to remove additional createPipeline step.
nandakumar131 commented on issue #663: HDDS-3156 update allocateContainer to remove additional createPipeline step. URL: https://github.com/apache/hadoop-ozone/pull/663#issuecomment-606771274 Thanks @timmylicheng for the patch and thanks @bharatviswa504 for the review. I will merge this shortly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #723: HDDS-3281. Add timeouts to all robot tests
adoroszlai commented on issue #723: HDDS-3281. Add timeouts to all robot tests URL: https://github.com/apache/hadoop-ozone/pull/723#issuecomment-606769748 > > @elek @hanishakoneru a 6-hours run despite timeout settings: https://github.com/apache/hadoop-ozone/runs/548358523 > > This branch was not rebased to include the timeout change. Can we rebase and retry. This run is a push build for 94413cd8c903d153ae183687b5cd4c5990aac341 on [`master`](https://github.com/apache/hadoop-ozone/commits/master), which does have the timeout change eece60420285330e21153d73d682c5eb3bc5458e. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2184) Rename ozone scmcli to ozone admin
[ https://issues.apache.org/jira/browse/HDDS-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDDS-2184. - Fix Version/s: 0.6.0 Resolution: Fixed Thanks [~elek] for the improvement and [~adoroszlai] for the reviews. > Rename ozone scmcli to ozone admin > -- > > Key: HDDS-2184 > URL: https://issues.apache.org/jira/browse/HDDS-2184 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Originally ozone scmcli designed to be used only by the developers. A very > cryptic name is chosen intentionally to frighten away the beginner users. > As we realized recently we started to use "ozone scmcli" as a generic admin > tool. More and more tools has been added which are useful not only for the > developers but for the administrators. > Therefore I suggest to rename "ozone scmcli" to something more meaningful. > For example to "ozone admin" > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] nandakumar131 commented on issue #643: HDDS-2339. Add OzoneManager to MiniOzoneChaosCluster
nandakumar131 commented on issue #643: HDDS-2339. Add OzoneManager to MiniOzoneChaosCluster URL: https://github.com/apache/hadoop-ozone/pull/643#issuecomment-606769075 Thanks @hanishakoneru for taking care of this. I don't have any major comments. We can merge it once @mukul1987 is ok with it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2184) Rename ozone scmcli to ozone admin
[ https://issues.apache.org/jira/browse/HDDS-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2184: - Labels: pull-request-available (was: ) > Rename ozone scmcli to ozone admin > -- > > Key: HDDS-2184 > URL: https://issues.apache.org/jira/browse/HDDS-2184 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > > Originally ozone scmcli designed to be used only by the developers. A very > cryptic name is chosen intentionally to frighten away the beginner users. > As we realized recently we started to use "ozone scmcli" as a generic admin > tool. More and more tools has been added which are useful not only for the > developers but for the administrators. > Therefore I suggest to rename "ozone scmcli" to something more meaningful. > For example to "ozone admin" > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] dineshchitlangia merged pull request #687: HDDS-2184. Rename ozone scmcli to ozone admin
dineshchitlangia merged pull request #687: HDDS-2184. Rename ozone scmcli to ozone admin URL: https://github.com/apache/hadoop-ozone/pull/687 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on issue #723: HDDS-3281. Add timeouts to all robot tests
hanishakoneru commented on issue #723: HDDS-3281. Add timeouts to all robot tests URL: https://github.com/apache/hadoop-ozone/pull/723#issuecomment-606761659 > @elek @hanishakoneru a 6-hours run despite timeout settings: https://github.com/apache/hadoop-ozone/runs/548358523 This branch was not rebased to include the timeout change. Can we rebase and retry. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on issue #723: HDDS-3281. Add timeouts to all robot tests
adoroszlai commented on issue #723: HDDS-3281. Add timeouts to all robot tests URL: https://github.com/apache/hadoop-ozone/pull/723#issuecomment-606754436 @elek @hanishakoneru a 6-hours run despite timeout settings: https://github.com/apache/hadoop-ozone/runs/548358523 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2621) Enable OM HA acceptance tests
[ https://issues.apache.org/jira/browse/HDDS-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru resolved HDDS-2621. -- Resolution: Fixed > Enable OM HA acceptance tests > - > > Key: HDDS-2621 > URL: https://issues.apache.org/jira/browse/HDDS-2621 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: om >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > OM HA robot tests were disable in HDDS-2533 as they were failing > intermittently. HDDS-2454 fixes some issues in the HA tests. Creating this > Jira so as to run re-enable the HA acceptance tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on issue #718: HDDS-3224. Enforce volume and bucket name rule at create time.
swagle commented on issue #718: HDDS-3224. Enforce volume and bucket name rule at create time. URL: https://github.com/apache/hadoop-ozone/pull/718#issuecomment-606749174 @ChenSammi wanted to get your input on above, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru merged pull request #265: HDDS-2621. Enable OM HA acceptance tests.
hanishakoneru merged pull request #265: HDDS-2621. Enable OM HA acceptance tests. URL: https://github.com/apache/hadoop-ozone/pull/265 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2621) Enable OM HA acceptance tests
[ https://issues.apache.org/jira/browse/HDDS-2621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2621: - Labels: pull-request-available (was: ) > Enable OM HA acceptance tests > - > > Key: HDDS-2621 > URL: https://issues.apache.org/jira/browse/HDDS-2621 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: om >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > OM HA robot tests were disable in HDDS-2533 as they were failing > intermittently. HDDS-2454 fixes some issues in the HA tests. Creating this > Jira so as to run re-enable the HA acceptance tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1554) Create disk tests for fault injection test
[ https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1554: Status: Open (was: Patch Available) > Create disk tests for fault injection test > -- > > Key: HDDS-1554 > URL: https://issues.apache.org/jira/browse/HDDS-1554 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1554.001.patch, HDDS-1554.002.patch, > HDDS-1554.003.patch, HDDS-1554.004.patch, HDDS-1554.005.patch, > HDDS-1554.006.patch, HDDS-1554.007.patch, HDDS-1554.008.patch, > HDDS-1554.009.patch, HDDS-1554.010.patch, HDDS-1554.011.patch, > HDDS-1554.012.patch, HDDS-1554.013.patch, HDDS-1554.014.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The current plan for fault injection disk tests are: > # Scenario 1 - Read/Write test > ## Run docker-compose to bring up a cluster > ## Initialize scm and om > ## Upload data to Ozone cluster > ## Verify data is correct > ## Shutdown cluster > # Scenario 2 - Read/Only test > ## Repeat Scenario 1 > ## Mount data disk as read only > ## Try to write data to Ozone cluster > ## Validate error message is correct > ## Shutdown cluster > # Scenario 3 - Corruption test > ## Repeat Scenario 2 > ## Shutdown cluster > ## Modify data disk data > ## Restart cluster > ## Validate error message for read from corrupted data > ## Validate error message for write to corrupted volume -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1554) Create disk tests for fault injection test
[ https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-1554: Labels: (was: pull-request-available) > Create disk tests for fault injection test > -- > > Key: HDDS-1554 > URL: https://issues.apache.org/jira/browse/HDDS-1554 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1554.001.patch, HDDS-1554.002.patch, > HDDS-1554.003.patch, HDDS-1554.004.patch, HDDS-1554.005.patch, > HDDS-1554.006.patch, HDDS-1554.007.patch, HDDS-1554.008.patch, > HDDS-1554.009.patch, HDDS-1554.010.patch, HDDS-1554.011.patch, > HDDS-1554.012.patch, HDDS-1554.013.patch, HDDS-1554.014.patch > > Time Spent: 40m > Remaining Estimate: 0h > > The current plan for fault injection disk tests are: > # Scenario 1 - Read/Write test > ## Run docker-compose to bring up a cluster > ## Initialize scm and om > ## Upload data to Ozone cluster > ## Verify data is correct > ## Shutdown cluster > # Scenario 2 - Read/Only test > ## Repeat Scenario 1 > ## Mount data disk as read only > ## Try to write data to Ozone cluster > ## Validate error message is correct > ## Shutdown cluster > # Scenario 3 - Corruption test > ## Repeat Scenario 2 > ## Shutdown cluster > ## Modify data disk data > ## Restart cluster > ## Validate error message for read from corrupted data > ## Validate error message for write to corrupted volume -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2301) Write path: Reduce read contention in rocksDB
[
https://issues.apache.org/jira/browse/HDDS-2301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated HDDS-2301:
Labels: performance (was: performance pull-request-available)
> Write path: Reduce read contention in rocksDB
> -
>
> Key: HDDS-2301
> URL: https://issues.apache.org/jira/browse/HDDS-2301
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 0.5.0
>Reporter: Rajesh Balamohan
>Assignee: Supratim Deka
>Priority: Major
> Labels: performance
> Attachments: om_write_profile.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Benchmark:
>
> Simple benchmark which creates 100 and 1000s of keys (empty directory) in
> OM. This is done in a tight loop and multiple threads from client side to add
> enough load on CPU. Note that intention is to understand the bottlenecks in
> OM (intentionally avoiding interactions with SCM & DN).
> Observation:
> -
> During write path, Ozone checks {{OMFileRequest.verifyFilesInPath}}. This
> internally calls {{omMetadataManager.getKeyTable().get(dbKeyName)}} for every
> write operation. This turns out to be expensive and chokes the write path.
> [https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java#L155]
> [https://github.com/apache/hadoop/blob/trunk/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java#L63]
> In most of the cases, directory creation would be fresh entry. In such cases,
> it would be good to try with {{RocksDB::keyMayExist.}}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3248) TestHddsDispatcher and TestHandler fails if run after other tests
[
https://issues.apache.org/jira/browse/HDDS-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh resolved HDDS-3248.
-
Fix Version/s: 0.6.0
Resolution: Fixed
> TestHddsDispatcher and TestHandler fails if run after other tests
> -
>
> Key: HDDS-3248
> URL: https://issues.apache.org/jira/browse/HDDS-3248
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Isa Hekmatizadeh
>Assignee: Isa Hekmatizadeh
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.6.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> TestHddsDispatcher and TestHandler fails if run after other tests with a
> error like this:
>
> {code:java}
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> StorageContainerMetrics already exists! at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.ozone.container.common.helpers.ContainerMetrics.create(ContainerMetrics.java:94)
> at
> org.apache.hadoop.ozone.container.common.impl.TestHddsDispatcher.createDispatcher(TestHddsDispatcher.java:271)
> at
> org.apache.hadoop.ozone.container.common.impl.TestHddsDispatcher.testCreateContainerWithWriteChunk(TestHddsDispatcher.java:156)
> ...
> {code}
> I think it's needed to call DefaultMetricsSystem.instance().shutdown()
> beforehand{color:#808080}
> {color}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3248) TestHddsDispatcher and TestHandler fails if run after other tests
[
https://issues.apache.org/jira/browse/HDDS-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17071935#comment-17071935
]
Mukul Kumar Singh commented on HDDS-3248:
-
Thanks for the contribution [~esa.hekmat] and [~pifta] for review. I have
merged this.
> TestHddsDispatcher and TestHandler fails if run after other tests
> -
>
> Key: HDDS-3248
> URL: https://issues.apache.org/jira/browse/HDDS-3248
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Isa Hekmatizadeh
>Assignee: Isa Hekmatizadeh
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> TestHddsDispatcher and TestHandler fails if run after other tests with a
> error like this:
>
> {code:java}
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> StorageContainerMetrics already exists! at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.ozone.container.common.helpers.ContainerMetrics.create(ContainerMetrics.java:94)
> at
> org.apache.hadoop.ozone.container.common.impl.TestHddsDispatcher.createDispatcher(TestHddsDispatcher.java:271)
> at
> org.apache.hadoop.ozone.container.common.impl.TestHddsDispatcher.testCreateContainerWithWriteChunk(TestHddsDispatcher.java:156)
> ...
> {code}
> I think it's needed to call DefaultMetricsSystem.instance().shutdown()
> beforehand{color:#808080}
> {color}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3248) TestHddsDispatcher and TestHandler fails if run after other tests
[
https://issues.apache.org/jira/browse/HDDS-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh reassigned HDDS-3248:
---
Assignee: Isa Hekmatizadeh
> TestHddsDispatcher and TestHandler fails if run after other tests
> -
>
> Key: HDDS-3248
> URL: https://issues.apache.org/jira/browse/HDDS-3248
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Isa Hekmatizadeh
>Assignee: Isa Hekmatizadeh
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> TestHddsDispatcher and TestHandler fails if run after other tests with a
> error like this:
>
> {code:java}
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> StorageContainerMetrics already exists! at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.ozone.container.common.helpers.ContainerMetrics.create(ContainerMetrics.java:94)
> at
> org.apache.hadoop.ozone.container.common.impl.TestHddsDispatcher.createDispatcher(TestHddsDispatcher.java:271)
> at
> org.apache.hadoop.ozone.container.common.impl.TestHddsDispatcher.testCreateContainerWithWriteChunk(TestHddsDispatcher.java:156)
> ...
> {code}
> I think it's needed to call DefaultMetricsSystem.instance().shutdown()
> beforehand{color:#808080}
> {color}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 merged pull request #704: HDDS-3248. shutdown defaultMetricsSystem before tests
mukul1987 merged pull request #704: HDDS-3248. shutdown defaultMetricsSystem before tests URL: https://github.com/apache/hadoop-ozone/pull/704 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2284) XceiverClientMetrics should be initialised as part of XceiverClientManager constructor
[ https://issues.apache.org/jira/browse/HDDS-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDDS-2284: Labels: (was: pull-request-available) > XceiverClientMetrics should be initialised as part of XceiverClientManager > constructor > -- > > Key: HDDS-2284 > URL: https://issues.apache.org/jira/browse/HDDS-2284 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > XceiverClientMetrics is currently initialized in the read write path, the > metric should be initialized while creating XceiverClientManager -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 closed pull request #236: HDDS-2284. XceiverClientMetrics should be initialised as part of XceiverClientManager constructor.
mukul1987 closed pull request #236: HDDS-2284. XceiverClientMetrics should be initialised as part of XceiverClientManager constructor. URL: https://github.com/apache/hadoop-ozone/pull/236 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant edited a comment on issue #578: HDDS-3053. Decrease the number of the chunk writer threads
bshashikant edited a comment on issue #578: HDDS-3053. Decrease the number of the chunk writer threads URL: https://github.com/apache/hadoop-ozone/pull/578#issuecomment-606416137 > Any more comments / suggestions? If i remember correctly, number of threads is currently for all pipelines a datanode has which may span over multiple disks. This is not a per disk configuration. I feel with more no of disks per datanode for write, we may need more threads depending upon the load. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2686) Use protobuf 3 instead of protobuf 2
[ https://issues.apache.org/jira/browse/HDDS-2686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17071831#comment-17071831 ] Wei-Chiu Chuang commented on HDDS-2686: --- Looks like this'll require Hadoop 3.3. Link HDDS-3292 to the jira. > Use protobuf 3 instead of protobuf 2 > > > Key: HDDS-2686 > URL: https://issues.apache.org/jira/browse/HDDS-2686 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Protobuf2 is 4.5 years old, Hadoop trunk already upgraded to use 3.x protobuf. > > Would be great to use recent protobuf version which can also provide > performance benefit and using new features. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on issue #716: HDDS-3155. Improved ozone client flush implementation to make it faster.
captainzmc commented on issue #716: HDDS-3155. Improved ozone client flush implementation to make it faster. URL: https://github.com/apache/hadoop-ozone/pull/716#issuecomment-606612561 @bshashikant Yes, I'll turn off the PR first. I will submit another PR and refresh the buffer with a timed task. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc closed pull request #716: HDDS-3155. Improved ozone client flush implementation to make it faster.
captainzmc closed pull request #716: HDDS-3155. Improved ozone client flush implementation to make it faster. URL: https://github.com/apache/hadoop-ozone/pull/716 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #734: HDDS-3240. Improve write efficiency by creating container in parallel
runzhiwang commented on a change in pull request #734: HDDS-3240. Improve write
efficiency by creating container in parallel
URL: https://github.com/apache/hadoop-ozone/pull/734#discussion_r400890248
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
##
@@ -664,12 +689,12 @@ private ByteString getCachedStateMachineData(Long
logIndex, long term,
future.complete(
getCachedStateMachineData(entry.getIndex(), entry.getTerm(),
requestProto));
- } catch (IOException e) {
+ } catch (Exception e) {
metrics.incNumReadStateMachineFails();
future.completeExceptionally(e);
}
return future;
-}, getChunkExecutor(requestProto.getWriteChunk()));
+}, getReadChunkExecutor(requestProto.getWriteChunk()));
Review comment:
@adoroszlai How do you think of @bshashikant 's advice ? I think it's also
reasonable, because lookup from cache is so fast, there is no need to do it in
another thread.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3307) Improve write efficiency by avoiding reverse DNS lookup
[ https://issues.apache.org/jira/browse/HDDS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3307: - Labels: pull-request-available (was: ) > Improve write efficiency by avoiding reverse DNS lookup > --- > > Key: HDDS-3307 > URL: https://issues.apache.org/jira/browse/HDDS-3307 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Major > Labels: pull-request-available > > [getHostName|https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java#L130] > cost about 15ms. It's also reported in > [JDK-6450279|https://bugs.openjdk.java.net/browse/JDK-6450279] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang opened a new pull request #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup
runzhiwang opened a new pull request #740: HDDS-3307. Improve write efficiency by avoiding reverse DNS lookup URL: https://github.com/apache/hadoop-ozone/pull/740 ## What changes were proposed in this pull request? **What's the problem ?** When create key, [getHostName()](https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java#L130) by IP need reverse DNS lookup, cost about 15ms. It's also reported in [JDK-6450279](https://bugs.openjdk.java.net/browse/JDK-6450279) **How to fix ?** client getLocalHost().getHostName() and add it in request, server get hostname from request. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3307 ## How was this patch tested? Existed UT and IT. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant commented on issue #716: HDDS-3155. Improved ozone client flush implementation to make it faster.
bshashikant commented on issue #716: HDDS-3155. Improved ozone client flush implementation to make it faster. URL: https://github.com/apache/hadoop-ozone/pull/716#issuecomment-606602244 I think there will be an alternate proposal to schedule a flush job over time rather tha doing a flush on specific data size boundary. Can we close this pr for now? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3249) TestContainerCache fails if runs after TestBlockDeletingService
[ https://issues.apache.org/jira/browse/HDDS-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-3249: -- Fix Version/s: 0.6.0 Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~esa.hekmat] for the contribution and [~pifta] for the discussions. I have committed this. > TestContainerCache fails if runs after TestBlockDeletingService > > > Key: HDDS-3249 > URL: https://issues.apache.org/jira/browse/HDDS-3249 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Isa Hekmatizadeh >Assignee: Isa Hekmatizadeh >Priority: Minor > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 40m > Remaining Estimate: 0h > > it fails on the line: > Assert.assertTrue(cache.isFull()); > because the ContainerCache.INSTANCE was created before with a different size > in TestBlockDeletingService. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bshashikant merged pull request #737: HDDS-3249: remove unrelated assert in ContainerCache
bshashikant merged pull request #737: HDDS-3249: remove unrelated assert in ContainerCache URL: https://github.com/apache/hadoop-ozone/pull/737 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #717: HDDS-3142. Create isolated enviornment for OM to test it without SCM (after revert)
elek closed pull request #717: HDDS-3142. Create isolated enviornment for OM to test it without SCM (after revert) URL: https://github.com/apache/hadoop-ozone/pull/717 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3236) Fix Dropwizard metrics mapping for latest Ratis metrics
[
https://issues.apache.org/jira/browse/HDDS-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek resolved HDDS-3236.
---
Target Version/s: 0.6.0
Resolution: Fixed
> Fix Dropwizard metrics mapping for latest Ratis metrics
> ---
>
> Key: HDDS-3236
> URL: https://issues.apache.org/jira/browse/HDDS-3236
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Marton Elek
>Assignee: Marton Elek
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Ratis use dropwizard metrics where the key parameters of the metrics (like
> group name or instance id) are part of the name of the metrics instead of
> using a tag.
> For example
> {code:java}
> ratis.log_appender.851cb00a-af97-455a-b079-d94a77d2a936@group-C14654DE8C2C.follower_65f881ea-8794-403d-be77-a030ed79c341_match_index
> {code}
> Instead of
> {code:java}
> ratis.log_appender_match_index{group="group-C14654DE8C2C",...} {code}
> It makes hard to combine the same metrics (match_index) from different
> sources.
> HDDS-2950 implemented a regexp based workaround, but the regexp doesn't match
> for the latest Ratis metrics.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #707: HDDS-3236. Fix Dropwizard metrics mapping for latest Ratis metrics.
elek closed pull request #707: HDDS-3236. Fix Dropwizard metrics mapping for latest Ratis metrics. URL: https://github.com/apache/hadoop-ozone/pull/707 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #719: HDDS-3270. Allow safemode listeners to be notified when some precheck rules pass
fapifta commented on a change in pull request #719: HDDS-3270. Allow safemode
listeners to be notified when some precheck rules pass
URL: https://github.com/apache/hadoop-ozone/pull/719#discussion_r400819244
##
File path:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
##
@@ -641,13 +663,26 @@ public boolean getSafeModeStatus() {
}
@Override
- public void handleSafeModeTransition(
+ public synchronized void handleSafeModeTransition(
SCMSafeModeManager.SafeModeStatus status) {
-this.isInSafeMode.set(status.getSafeModeStatus());
-if (!status.getSafeModeStatus()) {
- // TODO: #CLUTIL if we reenter safe mode the fixed interval pipeline
- // creation job needs to stop
+// TODO: #CLUTIL - handle safemode getting re-enabled
+boolean currentAllowPipelines =
+allowPipelineCreation.getAndSet(status.isPreCheckComplete());
+boolean currentlyInSafeMode =
+isInSafeMode.getAndSet(status.isInSafeMode());
+boolean triggerPipelines = false;
+
+// Trigger pipeline creation only if the preCheck status has changed to
+// complete.
+if (allowPipelineCreation.get() && !currentAllowPipelines) {
+ triggerPipelines = true;
+}
+// Start the pipeline creation thread only when safemode switches off
+if (!getSafeModeStatus() && currentlyInSafeMode) {
startPipelineCreator();
+ triggerPipelines = true;
+}
+if (triggerPipelines) {
Review comment:
Oh, I missed one small detail in the shouldSchedulePipelineCreator method...
and misinterpreted it sorry about that... So triggerPipelineCreation will
create pipelines if the safemode precheck is finished and allowPipelineCreation
is already true in SCMPipelineManager.
With that it makes perfect sense, that we want to trigger the pipeline
creation once when precheck is complete, and start the scheduled runs after the
safe mode is exited.
But in this case, it would be enough to call triggerPipelineCreation when we
arrived to the end of the precheck stage, and start the background pipeline
creator schedule when we left safe mode.
Now what I understand happens at a startup when this is interesting is the
following:
for (0..nodecount) {
triggerPipelineCreation() // noop as allowPipelineCreation is false
}
precheck finished
triggerPipelineCreation // runs and creates pipelines
safe mode exit
startFixedIntervalPipelineCreator // calls createPipelines right away once
at start
triggerPipelineCreation // tries to call createPipelines if it is not
running from the prev. call
I think we don't need this last call, which makes the triggerPipelines
boolean unnecessary and the code a bit simpler. Is it correct, or I am still
missing something? :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r400817019 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopDirGenerator.java ## @@ -0,0 +1,101 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.ozone.freon; + +import java.net.URI; +import java.util.concurrent.Callable; + +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdds.cli.HddsVersionProvider; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; + +import org.apache.commons.lang3.RandomStringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import picocli.CommandLine.Command; +import picocli.CommandLine.Option; + +/** + * Directory generator tool test om performance. + */ +@Command(name = "ddsg", +aliases = "dfs-directory-generator", +description = +"Create random directories to the any dfs compatible file system.", +versionProvider = HddsVersionProvider.class, +mixinStandardHelpOptions = true, +showDefaultValues = true) +public class HadoopDirGenerator extends BaseFreonGenerator Review comment: I have changed the class name to HadoopNestedDirGenerator This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a
freon generator to create nested directories.
URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r400817568
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopDirGenerator.java
##
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+package org.apache.hadoop.ozone.freon;
+
+import java.net.URI;
+import java.util.concurrent.Callable;
+
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+
+import org.apache.commons.lang3.RandomStringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Directory generator tool test om performance.
+ */
+@Command(name = "ddsg",
+aliases = "dfs-directory-generator",
+description =
+"Create random directories to the any dfs compatible file system.",
+versionProvider = HddsVersionProvider.class,
+mixinStandardHelpOptions = true,
+showDefaultValues = true)
+public class HadoopDirGenerator extends BaseFreonGenerator
+implements Callable {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(HadoopDirGenerator.class);
+
+ @Option(names = {"--dpath"},
+ description = "Hadoop FS directory system path",
+ defaultValue = "o3fs://bucket2.vol2")
+ private String rootPath;
+
+
+ @Option(names = {"-d", "--depth"},
+ description = "Number of directories to be generated recursively",
+ defaultValue = "5")
+ private int depth;
+
+ @Option(names = {"--span"},
+ description =
+ "Number of child directories to be created in leaf directory.",
+ defaultValue = "10")
+ private int span;
+
+ @Option(names = {"--nameLen"},
+ description =
+ "Length of the random name of directory you want to create.",
+ defaultValue = "10")
+ private int n;
+
+ private FileSystem fileSystem;
+
+ @Override
+ public Void call() throws Exception {
+
+init();
+OzoneConfiguration configuration = createOzoneConfiguration();
+fileSystem = FileSystem.get(URI.create(rootPath), configuration);
+runTests(this::createDir);
+return null;
+
+ }
+
+ private void createDir(long counter) throws Exception {
Review comment:
I have added the required comments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories.
aryangupta1998 commented on a change in pull request #730: HDDS-3289. Add a freon generator to create nested directories. URL: https://github.com/apache/hadoop-ozone/pull/730#discussion_r400817019 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopDirGenerator.java ## @@ -0,0 +1,101 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ +package org.apache.hadoop.ozone.freon; + +import java.net.URI; +import java.util.concurrent.Callable; + +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hdds.cli.HddsVersionProvider; +import org.apache.hadoop.hdds.conf.OzoneConfiguration; + +import org.apache.commons.lang3.RandomStringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import picocli.CommandLine.Command; +import picocli.CommandLine.Option; + +/** + * Directory generator tool test om performance. + */ +@Command(name = "ddsg", +aliases = "dfs-directory-generator", +description = +"Create random directories to the any dfs compatible file system.", +versionProvider = HddsVersionProvider.class, +mixinStandardHelpOptions = true, +showDefaultValues = true) +public class HadoopDirGenerator extends BaseFreonGenerator Review comment: I changed the name to HadoopNestedDirGenerator This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cku328 commented on issue #722: HDDS-3074. Make the configuration of container scrub consistent.
cku328 commented on issue #722: HDDS-3074. Make the configuration of container scrub consistent. URL: https://github.com/apache/hadoop-ozone/pull/722#issuecomment-606544142 Thanks @avijayanhwx & @bharatviswa504 for the review, and @elek for your advice on flaky test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3307) Improve write efficiency by avoiding reverse DNS lookup
[ https://issues.apache.org/jira/browse/HDDS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17071657#comment-17071657 ] runzhiwang commented on HDDS-3307: -- I'm working on it. > Improve write efficiency by avoiding reverse DNS lookup > --- > > Key: HDDS-3307 > URL: https://issues.apache.org/jira/browse/HDDS-3307 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Major > > [getHostName|https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java#L130] > cost about 15ms. It's also reported in > [JDK-6450279|https://bugs.openjdk.java.net/browse/JDK-6450279] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3307) Improve write efficiency by avoiding reverse DNS lookup
[ https://issues.apache.org/jira/browse/HDDS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3307: - Description: [getHostName|https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java#L130] cost about 15ms. It's also reported in [JDK-6450279|https://bugs.openjdk.java.net/browse/JDK-6450279] > Improve write efficiency by avoiding reverse DNS lookup > --- > > Key: HDDS-3307 > URL: https://issues.apache.org/jira/browse/HDDS-3307 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Major > > [getHostName|https://github.com/apache/hadoop-ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/OMClientRequest.java#L130] > cost about 15ms. It's also reported in > [JDK-6450279|https://bugs.openjdk.java.net/browse/JDK-6450279] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3023) Create Freon test to test isolated Ratis LEADER
[ https://issues.apache.org/jira/browse/HDDS-3023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3023. --- Target Version/s: 0.6.0 Resolution: Fixed > Create Freon test to test isolated Ratis LEADER > --- > > Key: HDDS-3023 > URL: https://issues.apache.org/jira/browse/HDDS-3023 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This is the pair/follow-up of HDDS-2974. It provides a new freon test which > can set up a new Ratis ring. If the Datanode was instrumented earlier to mock > leaders a real Ratis leader can be tested / measured without any overhead on > the leader side... -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #555: HDDS-3023. Create Freon test to test isolated Ratis LEADER
elek closed pull request #555: HDDS-3023. Create Freon test to test isolated Ratis LEADER URL: https://github.com/apache/hadoop-ozone/pull/555 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3307) Improve write efficiency by avoiding reverse DNS lookup
runzhiwang created HDDS-3307: Summary: Improve write efficiency by avoiding reverse DNS lookup Key: HDDS-3307 URL: https://issues.apache.org/jira/browse/HDDS-3307 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: runzhiwang Assignee: runzhiwang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3160) Disable index and filter block cache for RocksDB
[ https://issues.apache.org/jira/browse/HDDS-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3160. --- Fix Version/s: 0.6.0 Resolution: Fixed > Disable index and filter block cache for RocksDB > > > Key: HDDS-3160 > URL: https://issues.apache.org/jira/browse/HDDS-3160 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Marton Elek >Assignee: Marton Elek >Priority: Minor > Labels: pull-request-available > Fix For: 0.6.0 > > Attachments: key_allocation_after.png, key_allocation_before.png, > profile.png > > Time Spent: 10m > Remaining Estimate: 0h > > During preformance tests It was noticed that the OM performance is dropped > after 10-20 million of keys. (see the screenshot). > By default cache_index_and_filter_blocks is enabled for all of our RocksDB > instances (see DBProfile) which is not the best option. (For example see this > thread: https://github.com/facebook/rocksdb/issues/3961#) > With turning on this cache the indexes and bloom filters are cached **inside > the block cache** which makes slower the cache when we have significant data. > Without turning it on (based on my understanding) all the indexes will remain > open without any cache. With our current settings we have only a few number > of sst files (even with million of keys) therefore it seems to be safe to > turn this option off. > With turning this option of I was able to write >100M keys with high > throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
