[jira] [Assigned] (HDDS-3816) Erasure Coding in Apache Hadoop Ozone
[ https://issues.apache.org/jira/browse/HDDS-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDDS-3816: - Assignee: Uma Maheswara Rao G > Erasure Coding in Apache Hadoop Ozone > - > > Key: HDDS-3816 > URL: https://issues.apache.org/jira/browse/HDDS-3816 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: SCM >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: Erasure Coding in Apache Hadoop Ozone.pdf > > > We propose to implement Erasure Coding in Apache Hadoop Ozone to provide > efficient storage. With EC in place, Ozone can provide same or better > tolerance by giving 50% or more storage space savings. > In HDFS project, we already have native codecs(ISAL) and Java codecs > implemented, we can leverage the same or similar codec design. > However, the critical part of EC data layout design is in-progress, we will > post the design doc soon. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
GlenGeng commented on pull request #1228: URL: https://github.com/apache/hadoop-ozone/pull/1228#issuecomment-708134810 +1, LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
maobaolong commented on pull request #1228: URL: https://github.com/apache/hadoop-ozone/pull/1228#issuecomment-708134370 @GlenGeng @bharatviswa504 Thanks for your suggestion, I push a new commit to fix this. PTAL This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
maobaolong commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504378327
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String
key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
@GlenGeng @bharatviswa504 Thanks for your suggestion, I push a new
commit to fix this.
PTAL
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
GlenGeng commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504374249
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String
key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Hi @maobaolong, could you please add a new patch, just revert the change
and replace
```
} finally {
IOUtils.closeQuietly(ozoneOutputStream);
}
```
with
```
} finally {
if (ozoneOutputStream != null) {
ozoneOutputStream.close();
}
}
```
?
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4338) SCM web UI banner shows "HDFS SCM"
[ https://issues.apache.org/jira/browse/HDDS-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-4338: - Labels: newbie (was: ) > SCM web UI banner shows "HDFS SCM" > -- > > Key: HDDS-4338 > URL: https://issues.apache.org/jira/browse/HDDS-4338 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Wei-Chiu Chuang >Priority: Trivial > Labels: newbie > Attachments: Screen Shot 2020-10-12 at 6.42.31 PM.png > > > !Screen Shot 2020-10-12 at 6.42.31 PM.png! Let's call it Ozone SCM, shall we? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4342) Add DataNode state and transitions for a node going through upgrade
Prashant Pogde created HDDS-4342: Summary: Add DataNode state and transitions for a node going through upgrade Key: HDDS-4342 URL: https://issues.apache.org/jira/browse/HDDS-4342 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Prashant Pogde Assignee: Prashant Pogde -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4341) Bootstrap new OM node to single node OM cluster
[ https://issues.apache.org/jira/browse/HDDS-4341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-4341: - Description: Add a new OM node to an existing single node ratis enabled OM cluster. > Bootstrap new OM node to single node OM cluster > --- > > Key: HDDS-4341 > URL: https://issues.apache.org/jira/browse/HDDS-4341 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > > Add a new OM node to an existing single node ratis enabled OM cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4341) Bootstrap new OM node to single node OM cluster
Hanisha Koneru created HDDS-4341: Summary: Bootstrap new OM node to single node OM cluster Key: HDDS-4341 URL: https://issues.apache.org/jira/browse/HDDS-4341 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Hanisha Koneru Assignee: Hanisha Koneru -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4307) Start Trash Emptier in Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213327#comment-17213327 ] YiSheng Lien edited comment on HDDS-4307 at 10/13/20, 11:17 PM: Thank you for the reply. I would upload it ASAP. Update: Hi [~sadanand_shenoy], could you be so kind as to give me your slack name or email to me? (The design-doc is near complete, I would invite you to the doc if you don't mind, thanks) was (Author: cxorm): Thank you for the reply. I would upload it ASAP. > Start Trash Emptier in Ozone Manager > > > Key: HDDS-4307 > URL: https://issues.apache.org/jira/browse/HDDS-4307 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4314) OM Layout Version Manager init throws silent CNF error in integration tests.
[
https://issues.apache.org/jira/browse/HDDS-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4314:
Resolution: Fixed
Status: Resolved (was: Patch Available)
PR Merged.
> OM Layout Version Manager init throws silent CNF error in integration tests.
>
>
> Key: HDDS-4314
> URL: https://issues.apache.org/jira/browse/HDDS-4314
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> {code}
> org.reflections.ReflectionsException: could not get type for name
> mockit.MockUp
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312)
> at org.reflections.Reflections.expandSuperTypes(Reflections.java:382)
> at org.reflections.Reflections.(Reflections.java:140)
> at org.reflections.Reflections.(Reflections.java:182)
> at org.reflections.Reflections.(Reflections.java:155)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.registerOzoneManagerRequests(OMLayoutVersionManagerImpl.java:122)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.init(OMLayoutVersionManagerImpl.java:100)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.initialize(OMLayoutVersionManagerImpl.java:83)
> at org.apache.hadoop.ozone.om.OzoneManager.(OzoneManager.java:363)
> at
> org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:930)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.createOMService(MiniOzoneHAClusterImpl.java:379)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.build(MiniOzoneHAClusterImpl.java:294)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.init(TestOzoneManagerHA.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.lang.ClassNotFoundException: mockit.MockUp
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:310)
> ... 23 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx merged pull request #1492: HDDS-4314. OM Layout Version Manager init throws silent CNF error in integration tests.
avijayanhwx merged pull request #1492: URL: https://github.com/apache/hadoop-ozone/pull/1492 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4164) OM client request fails with "failed to commit as key is not found in OpenKey table"
[
https://issues.apache.org/jira/browse/HDDS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213426#comment-17213426
]
Wei-Chiu Chuang commented on HDDS-4164:
---
I hit this exact same issue. After patching my cluster with HDDS-4262, the bug
went away. So I think we're good to close this one.
> OM client request fails with "failed to commit as key is not found in OpenKey
> table"
>
>
> Key: HDDS-4164
> URL: https://issues.apache.org/jira/browse/HDDS-4164
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: OM HA
>Reporter: Lokesh Jain
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> OM client request fails with "failed to commit as key is not found in OpenKey
> table"
> {code:java}
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28868 $Proxy17.submitRequest over
> nodeId=om3,nodeAddress=vc1330.halxg.cloudera.com:9862
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28870 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28869 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28871 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28872 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28866 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28867 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28874 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28875 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 ERROR freon.BaseFreonGenerator: Error on executing task
> 14424
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to
> commit key, as /vol1/bucket1/akjkdz4hoj/14424/104766512182520809entry is not
> found in the OpenKey table
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:650)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:306)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:514)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:60)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKey$0(OzoneClientKeyGenerator.java:118)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:113)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:178)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:167)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:150)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4301) SCM CA certificate does not encode KeyUsage extension properly
[
https://issues.apache.org/jira/browse/HDDS-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vivek Ratnavel Subramanian updated HDDS-4301:
-
Summary: SCM CA certificate does not encode KeyUsage extension properly
(was: SCM CA certificate does not encode KeyUsage extension propertly)
> SCM CA certificate does not encode KeyUsage extension properly
> --
>
> Key: HDDS-4301
> URL: https://issues.apache.org/jira/browse/HDDS-4301
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Security
>Affects Versions: 1.0.0
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: pull-request-available
>
> This could be problematic with strict security provider such as FIPS. The
> default non-FIPS provider such as SunJCE and BC provider work fine though.
> This ticket is opened to fix it.
> {code:java}
> 2020-09-30 12:01:52,962 ERROR
> org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer:
> Unable to initialize CertificateServer.
> org.apache.hadoop.hdds.security.exception.SCMSecurityException:
> java.security.cert.CertificateParsingException: cannot construct KeyUsage:
> java.lang.IllegalArgumentException: illegal object in getInstance:
> com.safelogic.cryptocomply.asn1.DEROctetString
> at
> org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec.getPEMEncodedString(CertificateCodec.java:105)
> at
> org.apache.hadoop.hdds.security.x509.certificate.utils.CertificateCodec.writeCertificate(CertificateCodec.java:182)
> at
> org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.generateRootCertificate(DefaultCAServer.java:495)
> at
> org.apache.hadoop.hdds.security.x509.certificate.authority.DefaultCAServer.generateSelfSignedCA(DefaultCAServer.java:303)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4164) OM client request fails with "failed to commit as key is not found in OpenKey table"
[
https://issues.apache.org/jira/browse/HDDS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213347#comment-17213347
]
Bharat Viswanadham commented on HDDS-4164:
--
Annotation of methods is just for representational purpose, when I digged in to
Rpc code this annotation presence check is happened, but not used.
And also in OM Client protocol, we have only a single RPC method
*submitRequest*, so marking with @Idempotent/ @AT_MOST_ONCE is for
representational purpose, and it will not help in any way. So, planned to skip
marking the methods with annotation.
In NN, all methods which are non-idempotent, they use the cache, in ozone all
write requests use Ratis Retry cache, so it means all write requests use cache
by default.
> OM client request fails with "failed to commit as key is not found in OpenKey
> table"
>
>
> Key: HDDS-4164
> URL: https://issues.apache.org/jira/browse/HDDS-4164
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: OM HA
>Reporter: Lokesh Jain
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> OM client request fails with "failed to commit as key is not found in OpenKey
> table"
> {code:java}
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28868 $Proxy17.submitRequest over
> nodeId=om3,nodeAddress=vc1330.halxg.cloudera.com:9862
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28870 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28869 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28871 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28872 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28866 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28867 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28874 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28875 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 ERROR freon.BaseFreonGenerator: Error on executing task
> 14424
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to
> commit key, as /vol1/bucket1/akjkdz4hoj/14424/104766512182520809entry is not
> found in the OpenKey table
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:650)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:306)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:514)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:60)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKey$0(OzoneClientKeyGenerator.java:118)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:113)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:178)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:167)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:150)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
[GitHub] [hadoop-ozone] bharatviswa504 closed pull request #1418: HDDS-4209. S3A Filesystem does not work with Ozone S3.
bharatviswa504 closed pull request #1418: URL: https://github.com/apache/hadoop-ozone/pull/1418 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1418: HDDS-4209. S3A Filesystem does not work with Ozone S3.
bharatviswa504 commented on pull request #1418: URL: https://github.com/apache/hadoop-ozone/pull/1418#issuecomment-707983374 Closing this for now, as we donot support S3A FileSystem in filesystem path enabled mode. Need to document this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.
bharatviswa504 commented on pull request #1110: URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-707979758 I see with injection, we see error in logs but we don't see proper error code at the S3 client.  Not secure cluster, but tried with empty accesssKeyID and secret, still it is popping up with Bucket Not found. (Need to explore a bit how can we propagate this error) ``` s3g_1 | 2020-10-13 19:56:56,215 [qtp1296456465-16] ERROR s3.OzoneClientProducer: Error: s3g_1 | org.jboss.weld.exceptions.WeldException: WELD-49: Unable to invoke public void org.apache.hadoop.ozone.s3.AWSSignatureProcessor.init() throws java.lang.Exception on org.apache.hadoop.ozone.s3.AWSSignatureProcessor@32f12a54 s3g_1 |at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:99) s3g_1 |at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:80) s3g_1 |at org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122) s3g_1 |at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.postConstruct(CdiComponentProvider.java:887) s3g_1 |at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162) s3g_1 |at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96) s3g_1 |at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 |at org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.get(ContextualInstanceStrategy.java:177) s3g_1 |at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 |at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:99) s3g_1 |at org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:125) s3g_1 |at org.apache.hadoop.ozone.s3.AWSSignatureProcessor$Proxy$_$$_WeldClientProxy.getAwsAccessId(Unknown Source) s3g_1 |at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:79) s3g_1 |at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:68) s3g_1 |at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) s3g_1 |at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) s3g_1 |at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) s3g_1 |at java.base/java.lang.reflect.Method.invoke(Method.java:566) s3g_1 |at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) s3g_1 |at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) s3g_1 |at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) s3g_1 |at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) s3g_1 |at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) s3g_1 |at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) s3g_1 |at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) s3g_1 |at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) s3g_1 |at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) s3g_1 |at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) s3g_1 |at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) s3g_1 |at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) s3g_1 |at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) s3g_1 |at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) s3g_1 |at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) s3g_1 |at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) s3g_1 |at
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.
bharatviswa504 commented on pull request #1110: URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-707967571 @maobaolong Once we are in similar behavior with AWS S3 we can get this in. I agree that NPE does not look right here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4339) Ozone S3 gateway throws NPE with goofys
[ https://issues.apache.org/jira/browse/HDDS-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213335#comment-17213335 ] Bharat Viswanadham commented on HDDS-4339: -- This looks similar to HDDS-3843 where when auth header is missing we fail with NPE. [https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-648405855] We need to match with error code with AWS. Right now looks like we just print it. > Ozone S3 gateway throws NPE with goofys > --- > > Key: HDDS-4339 > URL: https://issues.apache.org/jira/browse/HDDS-4339 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Li Cheng >Priority: Blocker > Attachments: image-2020-10-13-15-23-49-864.png > > > Configured goofys and s3g on different hosts and Fiotest writes files on the > goofys mount point. Export AWS secrets on the s3g host. See a bunch of NPE in > s3g logs. > # Looks like missing AWS auth header could cause NPE. Looks like > AWSSignatureProcessor.init() doesn't handle header missing which causes NPE. > # Why it's missing AWS auth header is also unknown. > Note that there are files that have been successfully written into Ozone via > goofys, while not all of them are succeeded. > > 2020-10-13 11:18:43,425 [qtp1686100174-1238] ERROR > org.apache.hadoop.ozone.s3.OzoneClientProducer: Error: > org.jboss.weld.exceptions.WeldException: WELD-49: Unable to invoke public > void org.apache.hadoop.ozone.s3.AWSSignatureProcessor.init() throws > java.lang.Exception on > org.apache.hadoop.ozone.s3.AWSSignatureProcessor@5535155b > at > org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:99) > at > org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:80) > at > org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122) > at > org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.postConstruct(CdiComponentProvider.java:887) > at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162) > at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96) > at > org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) > at > org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.get(ContextualInstanceStrategy.java:177) > at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) > at > org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:99) > at > org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:125) > at > org.apache.hadoop.ozone.s3.AWSSignatureProcessor$Proxy$_$$_WeldClientProxy.getAwsAccessId(Unknown > Source) > at > org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:79) > at > org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:68) > at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) > at > org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) > at > org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) > at > org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) > at > org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) > at > org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) > at > org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) > at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) > at > org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) > at > org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) > at > org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) > at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) > at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) > at > org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) > at > org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) > at >
[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1110: HDDS-3843. Throw the specific exception other than NPE.
maobaolong opened a new pull request #1110: URL: https://github.com/apache/hadoop-ozone/pull/1110 ## What changes were proposed in this pull request? NPE shouldn't appear in the log. ## What is the link to the Apache JIRA HDDS-3843 ## How was this patch tested? Start a s3g without enable the security. Start a goofys without Authorization related setting. ls the dir within the mount point, the NPE will appear. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4307) Start Trash Emptier in Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213327#comment-17213327 ] YiSheng Lien commented on HDDS-4307: Thank you for the reply. I would upload it ASAP. > Start Trash Emptier in Ozone Manager > > > Key: HDDS-4307 > URL: https://issues.apache.org/jira/browse/HDDS-4307 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4307) Start Trash Emptier in Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212542#comment-17212542 ] YiSheng Lien edited comment on HDDS-4307 at 10/13/20, 7:29 PM: --- Hi [~sadanand_shenoy], thanks for this issue. I think this issue should be separated to several sub-tasks, and I'm working on the design-doc (I would upload to HDDS-2416 and here if you don't mind.) So I proposed that we could start the work til the uploading of design-doc (by this week), feel free to share your thoughts, thanks. was (Author: cxorm): Hi [~sadanand_shenoy], thanks for this issue. I think this issue should be separated to several sub-tasks, and I'm working on the design-doc (I would upload to HDDS-2416 and here if you don't mind.) So I supposed that we could start the work til the uploading of design-doc (by this week), feel free to share your thoughts, thanks. > Start Trash Emptier in Ozone Manager > > > Key: HDDS-4307 > URL: https://issues.apache.org/jira/browse/HDDS-4307 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4330) Bootstrap new OM node
[ https://issues.apache.org/jira/browse/HDDS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4330: - Labels: pull-request-available (was: ) > Bootstrap new OM node > - > > Key: HDDS-4330 > URL: https://issues.apache.org/jira/browse/HDDS-4330 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > > In a ratis enabled OM cluster, add support to bootstrap a new OM node and add > it to OM ratis ring. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru opened a new pull request #1494: HDDS-4330. Bootstrap new OM node
hanishakoneru opened a new pull request #1494: URL: https://github.com/apache/hadoop-ozone/pull/1494 ## What changes were proposed in this pull request? In a ratis enabled OM cluster, add support to bootstrap a new OM node and add it to OM ratis ring. First step would be to update the ozone-site.xml with the configs (nodeId, address, ports etc.) for the new OM. Note that we can only add one OM at a time. The new node should be started in BOOTSTRAP mode using the following command. This command will also initialize the OM. Hence, no need to run om init command before this command. `ozone om --bootstrap` After the new OM is bootstrapped, we need to run the following command. `ozone admin om addom -id= -nodeid=` This command will add the new OM to that ratis ring. The new will receive the snapshot and ratis logs from the leader and start participating in the ratis ring. TODO - This patch adds support to add a new OM to a single node ratis enabled OM cluster. To add a new OM to a multi OM cluster, we would have to send the updated OM information to all the follower OMs as well. This is required so that the followerOMs also can add the new OM to their peer information which is required in case a snapshot needs to be downloaded from the new OM. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4330 ## How was this patch tested? (Please explain how this patch was tested. Ex: unit tests, manual tests) (If this patch involves UI changes, please attach a screen-shot; otherwise, remove this) Will add tests in next iteration of the patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504192790
##
File path:
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String
key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Sorry @maobaolong for the analysis, I missed it.
Yes, it makes sense. Thank You @GlenGeng and @maobaolong for the detailed
analysis.
>How about we avoid call IOUtils.closeQuietly(ozoneOutputStream); to throw
out the >NO_SUCH_MULTIPART_UPLOAD_ERROR
I am +1 for this. In this way, we can return the proper error code when
during close if it happens.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4327) Potential resource leakage using BatchOperation
[ https://issues.apache.org/jira/browse/HDDS-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4327: - Labels: pull-request-available (was: ) > Potential resource leakage using BatchOperation > --- > > Key: HDDS-4327 > URL: https://issues.apache.org/jira/browse/HDDS-4327 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Bharat Viswanadham >Priority: Blocker > Labels: pull-request-available > > there are a number of places in the code where BatchOperation is used but not > closed. As a best practice, better to close them explicitly. > I have a stress test code that uses BatchOperation to insert into OM rocksdb. > Without closing BatchOperation explicitly, the process crashes after just a > few minutes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4327) Potential resource leakage using BatchOperation
[ https://issues.apache.org/jira/browse/HDDS-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-4327: - Status: Patch Available (was: In Progress) > Potential resource leakage using BatchOperation > --- > > Key: HDDS-4327 > URL: https://issues.apache.org/jira/browse/HDDS-4327 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Bharat Viswanadham >Priority: Blocker > > there are a number of places in the code where BatchOperation is used but not > closed. As a best practice, better to close them explicitly. > I have a stress test code that uses BatchOperation to insert into OM rocksdb. > Without closing BatchOperation explicitly, the process crashes after just a > few minutes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 opened a new pull request #1493: HDDS-4327. Potential resource leakage using BatchOperation.
bharatviswa504 opened a new pull request #1493: URL: https://github.com/apache/hadoop-ozone/pull/1493 ## What changes were proposed in this pull request? Potential resource leakage using BatchOperation. Use try enclosed resource/close the batch once after it usage is completed. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4327 ## How was this patch tested? Existing tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 removed a comment on pull request #1487: HDDS-4318. Disable single node pipeline creation by default in Ozone.
aryangupta1998 removed a comment on pull request #1487: URL: https://github.com/apache/hadoop-ozone/pull/1487#issuecomment-707740142 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4307) Start Trash Emptier in Ozone Manager
[ https://issues.apache.org/jira/browse/HDDS-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213294#comment-17213294 ] Sadanand Shenoy commented on HDDS-4307: --- Hi [~cxorm] ,thanks for your comment.As an initial implementation i am looking into using the existing Trash Emptier from Hadoop inside Ozone .Please upload your design if you have any. We can discuss about which approach would be better after that thanks. > Start Trash Emptier in Ozone Manager > > > Key: HDDS-4307 > URL: https://issues.apache.org/jira/browse/HDDS-4307 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1473: HDDS-4266: CreateFile : store parent dir entries into DirTable and file entry into separate FileTable
rakeshadr commented on a change in pull request #1473: URL: https://github.com/apache/hadoop-ozone/pull/1473#discussion_r504133459 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java ## @@ -466,27 +465,22 @@ public static void addDirectoryTableCacheEntries( * @param trxnLogIndex transaction log index * @return dbOmFileInfo, which keeps leaf node name in keyName field Review comment: Noted. Will take care in next PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4266) CreateFile : store parent dir entries into DirTable and file entry into separate FileTable
[
https://issues.apache.org/jira/browse/HDDS-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-4266:
-
Fix Version/s: 1.1.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> CreateFile : store parent dir entries into DirTable and file entry into
> separate FileTable
> --
>
> Key: HDDS-4266
> URL: https://issues.apache.org/jira/browse/HDDS-4266
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Rakesh Radhakrishnan
>Assignee: Rakesh Radhakrishnan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> This task is to handle #createFile ofs client request. Here the idea is to
> store all the missing parents in the {{keyname}} into 'DirTable' and file
> into 'FileTable'.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #1473: HDDS-4266: CreateFile : store parent dir entries into DirTable and file entry into separate FileTable
bharatviswa504 merged pull request #1473: URL: https://github.com/apache/hadoop-ozone/pull/1473 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1473: HDDS-4266: CreateFile : store parent dir entries into DirTable and file entry into separate FileTable
bharatviswa504 commented on a change in pull request #1473: URL: https://github.com/apache/hadoop-ozone/pull/1473#discussion_r504127124 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java ## @@ -466,27 +465,22 @@ public static void addDirectoryTableCacheEntries( * @param trxnLogIndex transaction log index * @return dbOmFileInfo, which keeps leaf node name in keyName field Review comment: Minor: There is no return from this method. Can be fixed in further jiras. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4314) OM Layout Version Manager init throws silent CNF error in integration tests.
[
https://issues.apache.org/jira/browse/HDDS-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4314:
Assignee: Aravindan Vijayan
Status: Patch Available (was: Open)
> OM Layout Version Manager init throws silent CNF error in integration tests.
>
>
> Key: HDDS-4314
> URL: https://issues.apache.org/jira/browse/HDDS-4314
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> {code}
> org.reflections.ReflectionsException: could not get type for name
> mockit.MockUp
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312)
> at org.reflections.Reflections.expandSuperTypes(Reflections.java:382)
> at org.reflections.Reflections.(Reflections.java:140)
> at org.reflections.Reflections.(Reflections.java:182)
> at org.reflections.Reflections.(Reflections.java:155)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.registerOzoneManagerRequests(OMLayoutVersionManagerImpl.java:122)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.init(OMLayoutVersionManagerImpl.java:100)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.initialize(OMLayoutVersionManagerImpl.java:83)
> at org.apache.hadoop.ozone.om.OzoneManager.(OzoneManager.java:363)
> at
> org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:930)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.createOMService(MiniOzoneHAClusterImpl.java:379)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.build(MiniOzoneHAClusterImpl.java:294)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.init(TestOzoneManagerHA.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.lang.ClassNotFoundException: mockit.MockUp
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:310)
> ... 23 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1492: HDDS-4314. OM Layout Version Manager init throws silent CNF error in integration tests.
avijayanhwx commented on pull request #1492: URL: https://github.com/apache/hadoop-ozone/pull/1492#issuecomment-707882048 @fapifta / @prashantpogde Please review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #1492: HDDS-4314. OM Layout Version Manager init throws silent CNF error in integration tests.
avijayanhwx opened a new pull request #1492: URL: https://github.com/apache/hadoop-ozone/pull/1492 ## What changes were proposed in this pull request? Fix class scanner initialization such that it does not scan test packages. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4314 ## How was this patch tested? Ran integration test without and with the fix. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4314) OM Layout Version Manager init throws silent CNF error in integration tests.
[
https://issues.apache.org/jira/browse/HDDS-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-4314:
-
Labels: pull-request-available (was: )
> OM Layout Version Manager init throws silent CNF error in integration tests.
>
>
> Key: HDDS-4314
> URL: https://issues.apache.org/jira/browse/HDDS-4314
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> {code}
> org.reflections.ReflectionsException: could not get type for name
> mockit.MockUp
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312)
> at org.reflections.Reflections.expandSuperTypes(Reflections.java:382)
> at org.reflections.Reflections.(Reflections.java:140)
> at org.reflections.Reflections.(Reflections.java:182)
> at org.reflections.Reflections.(Reflections.java:155)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.registerOzoneManagerRequests(OMLayoutVersionManagerImpl.java:122)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.init(OMLayoutVersionManagerImpl.java:100)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.initialize(OMLayoutVersionManagerImpl.java:83)
> at org.apache.hadoop.ozone.om.OzoneManager.(OzoneManager.java:363)
> at
> org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:930)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.createOMService(MiniOzoneHAClusterImpl.java:379)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.build(MiniOzoneHAClusterImpl.java:294)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.init(TestOzoneManagerHA.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.lang.ClassNotFoundException: mockit.MockUp
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:310)
> ... 23 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] rakeshadr commented on pull request #1473: HDDS-4266: CreateFile : store parent dir entries into DirTable and file entry into separate FileTable
rakeshadr commented on pull request #1473: URL: https://github.com/apache/hadoop-ozone/pull/1473#issuecomment-707874090 > Thank You for the updated patch. > I have one comment, (resolved fixed comments), rest LGTM. Thanks @bharatviswa504 for the detailed reviews. I've updated the patch addressing the comment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4122) Implement OM Delete Expired Open Key Request and Response
[ https://issues.apache.org/jira/browse/HDDS-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-4122. -- Fix Version/s: 1.1.0 Resolution: Fixed > Implement OM Delete Expired Open Key Request and Response > - > > Key: HDDS-4122 > URL: https://issues.apache.org/jira/browse/HDDS-4122 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM HA >Reporter: Ethan Rose >Assignee: Ethan Rose >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > Create an OM request and response that allows moving open keys from the open > key table to the deleted table in OM HA. The request portion of this > operation, which updates the open key table cache, will use a bucket lock. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #1435: HDDS-4122. Implement OM Delete Expired Open Key Request and Response
bharatviswa504 merged pull request #1435: URL: https://github.com/apache/hadoop-ozone/pull/1435 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1435: HDDS-4122. Implement OM Delete Expired Open Key Request and Response
bharatviswa504 commented on pull request #1435: URL: https://github.com/apache/hadoop-ozone/pull/1435#issuecomment-707853993 Thank You @errose28 for the contribution and @avijayanhwx for the review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4339) Ozone S3 gateway throws NPE with goofys
[
https://issues.apache.org/jira/browse/HDDS-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213186#comment-17213186
]
Bharat Viswanadham edited comment on HDDS-4339 at 10/13/20, 3:46 PM:
-
{quote}Export AWS secrets on the s3g host. See a bunch of NPE in s3g logs.
{quote}
#
{quote}Looks like missing AWS auth header could cause NPE. Looks like
AWSSignatureProcessor.init() doesn't handle header missing which causes NPE.
{quote} #
{quote}Why it's missing AWS auth header is also unknown.
{quote}
Exporting aws credentials should be done on the client node where we run
goofys? Is this the reason for auth header Null in S3G?
was (Author: bharatviswa):
{quote}Export AWS secrets on the s3g host. See a bunch of NPE in s3g logs.
{quote} #
{quote}Looks like missing AWS auth header could cause NPE. Looks like
AWSSignatureProcessor.init() doesn't handle header missing which causes
NPE.{quote}
#
{quote}Why it's missing AWS auth header is also unknown.{quote}
Exporting aws credentials should be done on the client node where we run goofys?
> Ozone S3 gateway throws NPE with goofys
> ---
>
> Key: HDDS-4339
> URL: https://issues.apache.org/jira/browse/HDDS-4339
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Li Cheng
>Priority: Blocker
> Attachments: image-2020-10-13-15-23-49-864.png
>
>
> Configured goofys and s3g on different hosts and Fiotest writes files on the
> goofys mount point. Export AWS secrets on the s3g host. See a bunch of NPE in
> s3g logs.
> # Looks like missing AWS auth header could cause NPE. Looks like
> AWSSignatureProcessor.init() doesn't handle header missing which causes NPE.
> # Why it's missing AWS auth header is also unknown.
> Note that there are files that have been successfully written into Ozone via
> goofys, while not all of them are succeeded.
>
> 2020-10-13 11:18:43,425 [qtp1686100174-1238] ERROR
> org.apache.hadoop.ozone.s3.OzoneClientProducer: Error:
> org.jboss.weld.exceptions.WeldException: WELD-49: Unable to invoke public
> void org.apache.hadoop.ozone.s3.AWSSignatureProcessor.init() throws
> java.lang.Exception on
> org.apache.hadoop.ozone.s3.AWSSignatureProcessor@5535155b
> at
> org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:99)
> at
> org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:80)
> at
> org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122)
> at
> org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.postConstruct(CdiComponentProvider.java:887)
> at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162)
> at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96)
> at
> org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
> at
> org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.get(ContextualInstanceStrategy.java:177)
> at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
> at
> org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:99)
> at
> org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:125)
> at
> org.apache.hadoop.ozone.s3.AWSSignatureProcessor$Proxy$_$$_WeldClientProxy.getAwsAccessId(Unknown
> Source)
> at
> org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:79)
> at
> org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:68)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88)
> at
> org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78)
> at
> org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100)
> at
> org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161)
> at
> org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180)
> at
> org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
> at
> org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
> at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
> at
>
[jira] [Commented] (HDDS-4339) Ozone S3 gateway throws NPE with goofys
[
https://issues.apache.org/jira/browse/HDDS-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213186#comment-17213186
]
Bharat Viswanadham commented on HDDS-4339:
--
{quote}Export AWS secrets on the s3g host. See a bunch of NPE in s3g logs.
{quote} #
{quote}Looks like missing AWS auth header could cause NPE. Looks like
AWSSignatureProcessor.init() doesn't handle header missing which causes
NPE.{quote}
#
{quote}Why it's missing AWS auth header is also unknown.{quote}
Exporting aws credentials should be done on the client node where we run goofys?
> Ozone S3 gateway throws NPE with goofys
> ---
>
> Key: HDDS-4339
> URL: https://issues.apache.org/jira/browse/HDDS-4339
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Li Cheng
>Priority: Blocker
> Attachments: image-2020-10-13-15-23-49-864.png
>
>
> Configured goofys and s3g on different hosts and Fiotest writes files on the
> goofys mount point. Export AWS secrets on the s3g host. See a bunch of NPE in
> s3g logs.
> # Looks like missing AWS auth header could cause NPE. Looks like
> AWSSignatureProcessor.init() doesn't handle header missing which causes NPE.
> # Why it's missing AWS auth header is also unknown.
> Note that there are files that have been successfully written into Ozone via
> goofys, while not all of them are succeeded.
>
> 2020-10-13 11:18:43,425 [qtp1686100174-1238] ERROR
> org.apache.hadoop.ozone.s3.OzoneClientProducer: Error:
> org.jboss.weld.exceptions.WeldException: WELD-49: Unable to invoke public
> void org.apache.hadoop.ozone.s3.AWSSignatureProcessor.init() throws
> java.lang.Exception on
> org.apache.hadoop.ozone.s3.AWSSignatureProcessor@5535155b
> at
> org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:99)
> at
> org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:80)
> at
> org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122)
> at
> org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.postConstruct(CdiComponentProvider.java:887)
> at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162)
> at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96)
> at
> org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
> at
> org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.get(ContextualInstanceStrategy.java:177)
> at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
> at
> org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:99)
> at
> org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:125)
> at
> org.apache.hadoop.ozone.s3.AWSSignatureProcessor$Proxy$_$$_WeldClientProxy.getAwsAccessId(Unknown
> Source)
> at
> org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:79)
> at
> org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:68)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88)
> at
> org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78)
> at
> org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100)
> at
> org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161)
> at
> org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180)
> at
> org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
> at
> org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
> at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
> at
> org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785)
> at
> org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885)
> at
> org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92)
> at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358)
> at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369)
> at
> org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70)
> at
>
[GitHub] [hadoop-ozone] flirmnave commented on a change in pull request #1490: HDDS-4269. Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS root directory.
flirmnave commented on a change in pull request #1490:
URL: https://github.com/apache/hadoop-ozone/pull/1490#discussion_r503964885
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/HddsVolumeUtil.java
##
@@ -212,6 +212,9 @@ public static boolean checkVolume(HddsVolume hddsVolume,
String scmId, String
} else {
// The hdds root dir should always have 2 files. One is Version file
// and other is SCM directory.
+ logger.error("The hdds root dir {} should always have 2 files. " +
+ "One is Version file and other is SCM directory.",
Review comment:
Thanks @jojochuang for reviewing and committing it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] github-actions[bot] commented on pull request #1487: HDDS-4318. Disable single node pipeline creation by default in Ozone.
github-actions[bot] commented on pull request #1487: URL: https://github.com/apache/hadoop-ozone/pull/1487#issuecomment-707740455 To re-run CI checks, please follow these steps with the source branch checked out: ``` git commit --allow-empty -m 'trigger new CI check' git push ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on pull request #1487: HDDS-4318. Disable single node pipeline creation by default in Ozone.
aryangupta1998 commented on pull request #1487: URL: https://github.com/apache/hadoop-ozone/pull/1487#issuecomment-707740142 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4308) Fix issue with quota update
[ https://issues.apache.org/jira/browse/HDDS-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YiSheng Lien reassigned HDDS-4308: -- Assignee: mingchao zhao > Fix issue with quota update > --- > > Key: HDDS-4308 > URL: https://issues.apache.org/jira/browse/HDDS-4308 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: mingchao zhao >Priority: Blocker > Labels: pull-request-available > > Currently volumeArgs using getCacheValue and put the same object in > doubleBuffer, this might cause issue. > Let's take the below scenario: > InitialVolumeArgs quotaBytes -> 1 > 1. T1 -> Update VolumeArgs, and subtracting 1000 and put this updated > volumeArgs to DoubleBuffer. > 2. T2-> Update VolumeArgs, and subtracting 2000 and has not still updated to > double buffer. > *Now at the end of flushing these transactions, our DB should have 7000 as > bytes used.* > Now T1 is picked by double Buffer and when it commits, and as it uses cached > Object put into doubleBuffer, it flushes to DB with the updated value from > T2(As it is a cache object) and update DB with bytesUsed as 7000. > And now OM has restarted, and only DB has transactions till T1. (We get this > info from TransactionInfo > Table(https://issues.apache.org/jira/browse/HDDS-3685) > Now T2 is again replayed, as it is not committed to DB, now DB will be again > subtracted with 2000, and now DB will have 5000. > But after T2, the value should be 7000, so we have DB in an incorrect state. > Issue here: > 1. As we use a cached object and put the same cached object into double > buffer this can cause this kind of issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] jojochuang commented on a change in pull request #1490: HDDS-4269. Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS root directory.
jojochuang commented on a change in pull request #1490:
URL: https://github.com/apache/hadoop-ozone/pull/1490#discussion_r503922135
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/utils/HddsVolumeUtil.java
##
@@ -212,6 +212,9 @@ public static boolean checkVolume(HddsVolume hddsVolume,
String scmId, String
} else {
// The hdds root dir should always have 2 files. One is Version file
// and other is SCM directory.
+ logger.error("The hdds root dir {} should always have 2 files. " +
+ "One is Version file and other is SCM directory.",
Review comment:
Let's add a more actionable message: "Please remove any other extra
files from the directory so that DataNode startup can proceed."
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel closed pull request #1461: Merge Master into HDDS-1880-Decom
sodonnel closed pull request #1461: URL: https://github.com/apache/hadoop-ozone/pull/1461 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel commented on pull request #1461: Merge Master into HDDS-1880-Decom
sodonnel commented on pull request #1461: URL: https://github.com/apache/hadoop-ozone/pull/1461#issuecomment-707693651 This PR was merged manually from the CLI to preserve the history. Therefore I am closing this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel opened a new pull request #1491: HDDS-4340. Add Operational State to the datanode list command
sodonnel opened a new pull request #1491: URL: https://github.com/apache/hadoop-ozone/pull/1491 ## What changes were proposed in this pull request? The existing CLI command `ozone admin datanode list` provides output like: ``` bash-4.2$ ozone admin datanode list Datanode: f2b2452a-bf7b-4c6d-b2d6-a0d9d219b21a (/default-rack/172.20.0.8/ozone_datanode_1.ozone_default/2 pipelines) Related pipelines: 16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Leader 4e45ff9c-478b-4ab8-a66c-7bfa98c8c632/ONE/RATIS/OPEN/Leader Datanode: 57c7fd5f-e32c-4de9-a04a-89d8d4273431 (/default-rack/172.20.0.6/ozone_datanode_3.ozone_default/2 pipelines) Related pipelines: 4b24bc61-28cf-471a-893c-a05cac273856/ONE/RATIS/OPEN/Leader 16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Follower ``` We should extend this to show the "Operational State" of the node for decommission, giving output like this: ``` bash-4.2$ ozone admin datanode list Datanode: cbd31d42-dc9d-4561-a800-2178fdb45866 (/default-rack/172.21.0.6/ozone_datanode_3.ozone_default/2 pipelines) Operational State: IN_SERVICE Related pipelines: a18df8cd-e868-43c5-9bee-90f547763090/THREE/RATIS/OPEN/Follower ec391cd9-cbc8-48f5-a569-9a161eab94ec/ONE/RATIS/OPEN/Leader Datanode: ebfedb58-05e2-4e0d-a95d-98589197c138 (/default-rack/172.21.0.8/ozone_datanode_1.ozone_default/2 pipelines) Operational State: IN_SERVICE Related pipelines: a18df8cd-e868-43c5-9bee-90f547763090/THREE/RATIS/OPEN/Leader 208952f5-f4d6-41b9-a487-3f7cf7ee8d8a/ONE/RATIS/OPEN/Leader Datanode: 044e64b3-120b-46c1-8cf6-94402f9eace5 (/default-rack/172.21.0.7/ozone_datanode_2.ozone_default/2 pipelines) Operational State: DECOMMISSIONING Related pipelines: a18df8cd-e868-43c5-9bee-90f547763090/THREE/RATIS/OPEN/Follower 9767ab14-303b-40da-bf4f-2e9588547509/ONE/RATIS/OPEN/Leader ``` ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4340 ## How was this patch tested? Verified the new output manually using a docker-compose cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4340) Add Operational State to the datanode list command
[
https://issues.apache.org/jira/browse/HDDS-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-4340:
-
Labels: pull-request-available (was: )
> Add Operational State to the datanode list command
> --
>
> Key: HDDS-4340
> URL: https://issues.apache.org/jira/browse/HDDS-4340
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: SCM Client
>Affects Versions: 1.1.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
>
> The existing CLI command `ozone admin datanode list` provides output like:
> {code}
> bash-4.2$ ozone admin datanode list
> Datanode: f2b2452a-bf7b-4c6d-b2d6-a0d9d219b21a
> (/default-rack/172.20.0.8/ozone_datanode_1.ozone_default/2 pipelines)
> Related pipelines:
> 16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Leader
> 4e45ff9c-478b-4ab8-a66c-7bfa98c8c632/ONE/RATIS/OPEN/Leader
> Datanode: 57c7fd5f-e32c-4de9-a04a-89d8d4273431
> (/default-rack/172.20.0.6/ozone_datanode_3.ozone_default/2 pipelines)
> Related pipelines:
> 4b24bc61-28cf-471a-893c-a05cac273856/ONE/RATIS/OPEN/Leader
> 16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Follower
> Datanode: 6699fc6d-5c2d-4110-8d88-5ffa5b99f326
> (/default-rack/172.20.0.3/ozone_datanode_2.ozone_default/2 pipelines)
> Related pipelines:
> 16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Follower
> 5ce21cae-9a2d-486d-8b4b-f8ddf75efc61/ONE/RATIS/OPEN/Leader
> {code}
> We should extend this to show the "Operational State" of the node for
> decommission.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4340) Add Operational State to the datanode list command
Stephen O'Donnell created HDDS-4340:
---
Summary: Add Operational State to the datanode list command
Key: HDDS-4340
URL: https://issues.apache.org/jira/browse/HDDS-4340
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Components: SCM Client
Affects Versions: 1.1.0
Reporter: Stephen O'Donnell
Assignee: Stephen O'Donnell
The existing CLI command `ozone admin datanode list` provides output like:
{code}
bash-4.2$ ozone admin datanode list
Datanode: f2b2452a-bf7b-4c6d-b2d6-a0d9d219b21a
(/default-rack/172.20.0.8/ozone_datanode_1.ozone_default/2 pipelines)
Related pipelines:
16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Leader
4e45ff9c-478b-4ab8-a66c-7bfa98c8c632/ONE/RATIS/OPEN/Leader
Datanode: 57c7fd5f-e32c-4de9-a04a-89d8d4273431
(/default-rack/172.20.0.6/ozone_datanode_3.ozone_default/2 pipelines)
Related pipelines:
4b24bc61-28cf-471a-893c-a05cac273856/ONE/RATIS/OPEN/Leader
16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Follower
Datanode: 6699fc6d-5c2d-4110-8d88-5ffa5b99f326
(/default-rack/172.20.0.3/ozone_datanode_2.ozone_default/2 pipelines)
Related pipelines:
16561bc4-746a-4c79-b6f8-1c275b31e37d/THREE/RATIS/OPEN/Follower
5ce21cae-9a2d-486d-8b4b-f8ddf75efc61/ONE/RATIS/OPEN/Leader
{code}
We should extend this to show the "Operational State" of the node for
decommission.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3103) Have multi-raft pipeline calculator to recommend best pipeline number per datanode
[ https://issues.apache.org/jira/browse/HDDS-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213047#comment-17213047 ] Shashikant Banerjee commented on HDDS-3103: --- This should have been addresed with HDDS-3700. > Have multi-raft pipeline calculator to recommend best pipeline number per > datanode > -- > > Key: HDDS-3103 > URL: https://issues.apache.org/jira/browse/HDDS-3103 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Affects Versions: 0.5.0 >Reporter: Li Cheng >Priority: Critical > > PipelinePlacementPolicy should have a calculator method to recommend better > number for pipeline number per node. The number used to come from > ozone.datanode.pipeline.limit in config. SCM should be able to consider how > many ratis dir and the ratis retry timeout to recommend the best pipeline > number for every node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4164) OM client request fails with "failed to commit as key is not found in OpenKey table"
[
https://issues.apache.org/jira/browse/HDDS-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213034#comment-17213034
]
Lokesh Jain commented on HDDS-4164:
---
[~bharat] We had also discussed
https://issues.apache.org/jira/browse/HDDS-3580. HDDS-4262 would also require
HDDS-3580, right?
> OM client request fails with "failed to commit as key is not found in OpenKey
> table"
>
>
> Key: HDDS-4164
> URL: https://issues.apache.org/jira/browse/HDDS-4164
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: OM HA
>Reporter: Lokesh Jain
>Assignee: Bharat Viswanadham
>Priority: Blocker
>
> OM client request fails with "failed to commit as key is not found in OpenKey
> table"
> {code:java}
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28868 $Proxy17.submitRequest over
> nodeId=om3,nodeAddress=vc1330.halxg.cloudera.com:9862
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28870 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:53 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28869 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28871 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28872 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28866 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28867 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28874 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 WARN retry.RetryInvocationHandler: A failover has occurred
> since the start of call #28875 $Proxy17.submitRequest over
> nodeId=om1,nodeAddress=vc1325.halxg.cloudera.com:9862
> 20/08/28 03:21:54 ERROR freon.BaseFreonGenerator: Error on executing task
> 14424
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to
> commit key, as /vol1/bucket1/akjkdz4hoj/14424/104766512182520809entry is not
> found in the OpenKey table
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:593)
> at
> org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:650)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:306)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:514)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:60)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKey$0(OzoneClientKeyGenerator.java:118)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:113)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:178)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:167)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:150)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4336) ContainerInfo does not persist BCSID leading to failed replicas reports
[
https://issues.apache.org/jira/browse/HDDS-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen O'Donnell updated HDDS-4336:
Fix Version/s: 1.1.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ContainerInfo does not persist BCSID leading to failed replicas reports
> ---
>
> Key: HDDS-4336
> URL: https://issues.apache.org/jira/browse/HDDS-4336
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 1.1.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> If you create a container, and then close it, the BCSID is synced on the
> datanodes and then the value is updated in SCM via setting the "sequenceID"
> field on the containerInfo object for the container.
> If you later restart just SCM, the sequenceID becomes zero, and then
> container reports for the replica fail with a stack trace like:
> {code}
> Exception in thread "EventQueue-ContainerReportForContainerReportHandler"
> java.lang.AssertionError
> at
> org.apache.hadoop.hdds.scm.container.ContainerInfo.updateSequenceId(ContainerInfo.java:176)
> at
> org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.updateContainerStats(AbstractContainerReportHandler.java:108)
> at
> org.apache.hadoop.hdds.scm.container.AbstractContainerReportHandler.processContainerReplica(AbstractContainerReportHandler.java:83)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:162)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:130)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:50)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:81)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> The assertion here is failing, as it does not allow for the sequenceID to be
> changed on a CLOSED container:
> {code}
> public void updateSequenceId(long sequenceID) {
> assert (isOpen() || state == HddsProtos.LifeCycleState.QUASI_CLOSED);
> sequenceId = max(sequenceID, sequenceId);
> }
> {code}
> The issue seems to be caused by the serialisation and deserialisation of the
> containerInfo object to protobuf, as sequenceId never persisted or restored.
> However, I am also confused about how this ever worked, as this is a pretty
> significant problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] sodonnel merged pull request #1488: HDDS-4336. ContainerInfo does not persist BCSID (sequenceId) leading to failed replica reports
sodonnel merged pull request #1488: URL: https://github.com/apache/hadoop-ozone/pull/1488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4269) Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS root directory
[
https://issues.apache.org/jira/browse/HDDS-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-4269:
-
Labels: newbie pull-request-available (was: newbie)
> Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS
> root directory
>
>
> Key: HDDS-4269
> URL: https://issues.apache.org/jira/browse/HDDS-4269
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 1.1.0
>Reporter: Wei-Chiu Chuang
>Assignee: Zheng Huang-Mu
>Priority: Major
> Labels: newbie, pull-request-available
>
> Took me some time to debug a trivial bug.
> DataNode crashes after this mysterious error and no explanation:
> {noformat}
> 10:11:44.382 PM INFOMutableVolumeSetMoving Volume :
> /var/lib/hadoop-ozone/fake_datanode/data/hdds to failed Volumes
> 10:11:46.287 PM ERROR StateContextCritical error occurred in
> StateMachine, setting shutDownMachine
> 10:11:46.287 PM ERROR DatanodeStateMachineDatanodeStateMachine
> Shutdown due to an critical error
> {noformat}
> Turns out that if there are unexpected files under the hdds directory
> ($hdds.datanode.dir/hdds), DN thinks the volume is bad and move it to failed
> volume list, without an error explanation. I was editing the VERSION file and
> vim created a temp file under the directory. This is impossible to debug
> without reading the code.
> {code:java|title=HddsVolumeUtil#checkVolume()}
> } else if(hddsFiles.length == 2) {
> // The files should be Version and SCM directory
> if (scmDir.exists()) {
> return true;
> } else {
> logger.error("Volume {} is in Inconsistent state, expected scm " +
> "directory {} does not exist", volumeRoot, scmDir
> .getAbsolutePath());
> return false;
> }
> } else {
> // The hdds root dir should always have 2 files. One is Version file
> // and other is SCM directory.
> < HERE!
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] flirmnave opened a new pull request #1490: HDDS-4269. Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS root directory.
flirmnave opened a new pull request #1490: URL: https://github.com/apache/hadoop-ozone/pull/1490 ## What changes were proposed in this pull request? Adding a log makes it easy to track the problem. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4269 ## How was this patch tested? Test on local. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4269) Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS root directory
[
https://issues.apache.org/jira/browse/HDDS-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Huang-Mu reassigned HDDS-4269:
Assignee: Zheng Huang-Mu
> Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS
> root directory
>
>
> Key: HDDS-4269
> URL: https://issues.apache.org/jira/browse/HDDS-4269
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 1.1.0
>Reporter: Wei-Chiu Chuang
>Assignee: Zheng Huang-Mu
>Priority: Major
> Labels: newbie
>
> Took me some time to debug a trivial bug.
> DataNode crashes after this mysterious error and no explanation:
> {noformat}
> 10:11:44.382 PM INFOMutableVolumeSetMoving Volume :
> /var/lib/hadoop-ozone/fake_datanode/data/hdds to failed Volumes
> 10:11:46.287 PM ERROR StateContextCritical error occurred in
> StateMachine, setting shutDownMachine
> 10:11:46.287 PM ERROR DatanodeStateMachineDatanodeStateMachine
> Shutdown due to an critical error
> {noformat}
> Turns out that if there are unexpected files under the hdds directory
> ($hdds.datanode.dir/hdds), DN thinks the volume is bad and move it to failed
> volume list, without an error explanation. I was editing the VERSION file and
> vim created a temp file under the directory. This is impossible to debug
> without reading the code.
> {code:java|title=HddsVolumeUtil#checkVolume()}
> } else if(hddsFiles.length == 2) {
> // The files should be Version and SCM directory
> if (scmDir.exists()) {
> return true;
> } else {
> logger.error("Volume {} is in Inconsistent state, expected scm " +
> "directory {} does not exist", volumeRoot, scmDir
> .getAbsolutePath());
> return false;
> }
> } else {
> // The hdds root dir should always have 2 files. One is Version file
> // and other is SCM directory.
> < HERE!
> return false;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4308) Fix issue with quota update
[ https://issues.apache.org/jira/browse/HDDS-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4308: - Labels: pull-request-available (was: ) > Fix issue with quota update > --- > > Key: HDDS-4308 > URL: https://issues.apache.org/jira/browse/HDDS-4308 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Priority: Blocker > Labels: pull-request-available > > Currently volumeArgs using getCacheValue and put the same object in > doubleBuffer, this might cause issue. > Let's take the below scenario: > InitialVolumeArgs quotaBytes -> 1 > 1. T1 -> Update VolumeArgs, and subtracting 1000 and put this updated > volumeArgs to DoubleBuffer. > 2. T2-> Update VolumeArgs, and subtracting 2000 and has not still updated to > double buffer. > *Now at the end of flushing these transactions, our DB should have 7000 as > bytes used.* > Now T1 is picked by double Buffer and when it commits, and as it uses cached > Object put into doubleBuffer, it flushes to DB with the updated value from > T2(As it is a cache object) and update DB with bytesUsed as 7000. > And now OM has restarted, and only DB has transactions till T1. (We get this > info from TransactionInfo > Table(https://issues.apache.org/jira/browse/HDDS-3685) > Now T2 is again replayed, as it is not committed to DB, now DB will be again > subtracted with 2000, and now DB will have 5000. > But after T2, the value should be 7000, so we have DB in an incorrect state. > Issue here: > 1. As we use a cached object and put the same cached object into double > buffer this can cause this kind of issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc opened a new pull request #1489: HDDS-4308. Fix issue with quota update
captainzmc opened a new pull request #1489: URL: https://github.com/apache/hadoop-ozone/pull/1489 ## What changes were proposed in this pull request? Currently volumeArgs using getCacheValue and put the same object in doubleBuffer, this might cause issue. Let's take the below scenario: InitialVolumeArgs quotaBytes -> 1 1. T1 -> Update VolumeArgs, and subtracting 1000 and put this updated volumeArgs to DoubleBuffer. 2. T2-> Update VolumeArgs, and subtracting 2000 and has not still updated to double buffer. Now at the end of flushing these transactions, our DB should have 7000 as bytes used. Now T1 is picked by double Buffer and when it commits, and as it uses cached Object put into doubleBuffer, it flushes to DB with the updated value from T2(As it is a cache object) and update DB with bytesUsed as 7000. And now OM has restarted, and only DB has transactions till T1. (We get this info from TransactionInfo Table(https://issues.apache.org/jira/browse/HDDS-3685) Now T2 is again replayed, as it is not committed to DB, now DB will be again subtracted with 2000, and now DB will have 5000. But after T2, the value should be 7000, so we have DB in an incorrect state. Issue here: 1. As we use a cached object and put the same cached object into double buffer this can cause this kind of issue. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-4308 ## How was this patch tested? Use the current UT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2411) Create DataChunkValidator Freon test
[ https://issues.apache.org/jira/browse/HDDS-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] François Risch resolved HDDS-2411. -- Resolution: Fixed > Create DataChunkValidator Freon test > > > Key: HDDS-2411 > URL: https://issues.apache.org/jira/browse/HDDS-2411 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: freon >Reporter: Marton Elek >Assignee: François Risch >Priority: Major > Labels: newbie, pull-request-available > > HDDS-2327 introduced a new load test which generates a lot of WriteChunk > request. > As with other freon test (for example with. > HadoopFsGenerator/HadoopFsValidator) we need an other load test for > validation/read path. > It should be almost the same DatanodeChunkGenerator but it should read the > first chunk and compare all the others (very similar to the HadoopFsValidator > or OzoneClientKeyValidator) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4339) Ozone S3 gateway throws NPE with goofys
Li Cheng created HDDS-4339: -- Summary: Ozone S3 gateway throws NPE with goofys Key: HDDS-4339 URL: https://issues.apache.org/jira/browse/HDDS-4339 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 1.0.0 Reporter: Li Cheng Attachments: image-2020-10-13-15-23-49-864.png Configured goofys and s3g on different hosts and Fiotest writes files on the goofys mount point. Export AWS secrets on the s3g host. See a bunch of NPE in s3g logs. # Looks like missing AWS auth header could cause NPE. Looks like AWSSignatureProcessor.init() doesn't handle header missing which causes NPE. # Why it's missing AWS auth header is also unknown. Note that there are files that have been successfully written into Ozone via goofys, while not all of them are succeeded. 2020-10-13 11:18:43,425 [qtp1686100174-1238] ERROR org.apache.hadoop.ozone.s3.OzoneClientProducer: Error: org.jboss.weld.exceptions.WeldException: WELD-49: Unable to invoke public void org.apache.hadoop.ozone.s3.AWSSignatureProcessor.init() throws java.lang.Exception on org.apache.hadoop.ozone.s3.AWSSignatureProcessor@5535155b at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:99) at org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:80) at org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122) at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.postConstruct(CdiComponentProvider.java:887) at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:162) at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96) at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) at org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.get(ContextualInstanceStrategy.java:177) at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:99) at org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:125) at org.apache.hadoop.ozone.s3.AWSSignatureProcessor$Proxy$_$$_WeldClientProxy.getAwsAccessId(Unknown Source) at org.apache.hadoop.ozone.s3.OzoneClientProducer.getClient(OzoneClientProducer.java:79) at org.apache.hadoop.ozone.s3.OzoneClientProducer.createClient(OzoneClientProducer.java:68) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88) at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78) at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100) at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785) at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885) at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92) at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358) at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369) at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70) at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48) at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72) at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117) at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:873) at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159) at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70) at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) at
[jira] [Assigned] (HDDS-3103) Have multi-raft pipeline calculator to recommend best pipeline number per datanode
[ https://issues.apache.org/jira/browse/HDDS-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Cheng reassigned HDDS-3103: -- Assignee: (was: Li Cheng) > Have multi-raft pipeline calculator to recommend best pipeline number per > datanode > -- > > Key: HDDS-3103 > URL: https://issues.apache.org/jira/browse/HDDS-3103 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Affects Versions: 0.5.0 >Reporter: Li Cheng >Priority: Critical > > PipelinePlacementPolicy should have a calculator method to recommend better > number for pipeline number per node. The number used to come from > ozone.datanode.pipeline.limit in config. SCM should be able to consider how > many ratis dir and the ratis retry timeout to recommend the best pipeline > number for every node. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
