[jira] [Assigned] (HADOOP-17680) Allow ProtobufRpcEngine to be extensible
[ https://issues.apache.org/jira/browse/HADOOP-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri reassigned HADOOP-17680: - Component/s: (was: hdfs) common Key: HADOOP-17680 (was: HDFS-15912) Assignee: (was: Hector Sandoval Chaverri) Project: Hadoop Common (was: Hadoop HDFS) > Allow ProtobufRpcEngine to be extensible > > > Key: HADOOP-17680 > URL: https://issues.apache.org/jira/browse/HADOOP-17680 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The ProtobufRpcEngine class doesn't allow for new RpcEngine implementations > to extend some of its inner classes (e.g. Invoker and > Server.ProtoBufRpcInvoker). Also, some of its methods are long enough such > that overriding them would result in a lot of code duplication (e.g. > Invoker#invoke and Server.ProtoBufRpcInvoker#call). > When implementing a new RpcEngine, it would be helpful to reuse most of the > code already in ProtobufRpcEngine. This would allow new fields to be added to > the RPC header or message with minimal code changes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17680) Allow ProtobufRpcEngine to be extensible
[ https://issues.apache.org/jira/browse/HADOOP-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17338645#comment-17338645 ] Hector Sandoval Chaverri commented on HADOOP-17680: --- Thanks for taking a look [~shv]. I've made the following changes: # Moved this Jira to the Hadoop Common project. # Reverted the changes to make members protected and only added the getters that are needed. There are still warnings about pre-existing issues, regarding the number of parameters in the Invoker constructor. > Allow ProtobufRpcEngine to be extensible > > > Key: HADOOP-17680 > URL: https://issues.apache.org/jira/browse/HADOOP-17680 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > The ProtobufRpcEngine class doesn't allow for new RpcEngine implementations > to extend some of its inner classes (e.g. Invoker and > Server.ProtoBufRpcInvoker). Also, some of its methods are long enough such > that overriding them would result in a lot of code duplication (e.g. > Invoker#invoke and Server.ProtoBufRpcInvoker#call). > When implementing a new RpcEngine, it would be helpful to reuse most of the > code already in ProtobufRpcEngine. This would allow new fields to be added to > the RPC header or message with minimal code changes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17819) Add extensions to ProtobufRpcEngine RequestHeaderProto
Hector Sandoval Chaverri created HADOOP-17819: - Summary: Add extensions to ProtobufRpcEngine RequestHeaderProto Key: HADOOP-17819 URL: https://issues.apache.org/jira/browse/HADOOP-17819 Project: Hadoop Common Issue Type: Improvement Components: common Reporter: Hector Sandoval Chaverri The header used in ProtobufRpcEngine messages doesn't allow for new properties to be added by child classes. We can add a range of extensions that can be useful for proto classes that need to extend RequestHeaderProto. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18167) Add metrics to track delegation token secret manager operations
Hector Sandoval Chaverri created HADOOP-18167: - Summary: Add metrics to track delegation token secret manager operations Key: HADOOP-18167 URL: https://issues.apache.org/jira/browse/HADOOP-18167 Project: Hadoop Common Issue Type: Improvement Reporter: Hector Sandoval Chaverri New metrics to track operations that store, update and remove delegation tokens in implementations of AbstractDelegationTokenSecretManager. This will help evaluate the impact of using different secret managers and add optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510910#comment-17510910 ] Hector Sandoval Chaverri commented on HADOOP-18167: --- Hi [~ste...@apache.org], I saw that a few classes , such as S3AInstrumentation, implement IOStatisticsSource and use IOStatisticsStore to track different counters. Is this the approach that you think we should follow? Could you also help explain what's the consumer of the IOStatistics? > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18167: -- Attachment: HADOOP-18167-branch-2.10.patch Status: Patch Available (was: Open) > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10.patch > > Time Spent: 20m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18167: -- Attachment: HADOOP-18167-branch-2.10-2.patch > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522530#comment-17522530 ] Hector Sandoval Chaverri commented on HADOOP-18167: --- [~fengnanli] / [~jing9] / [~inigoiri] Would you be able to help review this, since Owen is out for next week? Thank you! > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18167: -- Attachment: HADOOP-18167-branch-2.10-3.patch > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10-3.patch, HADOOP-18167-branch-2.10.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18167: -- Attachment: HADOOP-18167-branch-3.3.patch > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10-3.patch, HADOOP-18167-branch-2.10.patch, > HADOOP-18167-branch-3.3.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18167: -- Attachment: HADOOP-18167-branch-2.10-4.patch > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10-3.patch, HADOOP-18167-branch-2.10-4.patch, > HADOOP-18167-branch-2.10.patch, HADOOP-18167-branch-3.3.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18222) Prevent DelegationTokenSecretManagerMetrics from registering multiple times
Hector Sandoval Chaverri created HADOOP-18222:
-
Summary: Prevent DelegationTokenSecretManagerMetrics from
registering multiple times
Key: HADOOP-18222
URL: https://issues.apache.org/jira/browse/HADOOP-18222
Project: Hadoop Common
Issue Type: Improvement
Reporter: Hector Sandoval Chaverri
After committing HADOOP-18167, we received reports of the following error when
ResourceManager is initialized:
{noformat}
Caused by: java.io.IOException: Problem starting http server
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1389)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:475)
... 4 more
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
DelegationTokenSecretManagerMetrics already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
at
org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:71)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenSecretManagerMetrics.create(AbstractDelegationTokenSecretManager.java:878)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.(AbstractDelegationTokenSecretManager.java:152)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$DelegationTokenSecretManager.(DelegationTokenManager.java:72)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.(DelegationTokenManager.java:122)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:161)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:130)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:214)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:180)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53){noformat}
This can happen if MetricsSystemImpl#init is called and multiple metrics are
registered with the same name. A proposed solution is to declare the metrics in
AbstractDelegationTokenSecretManager as singleton, which would prevent multiple
instances DelegationTokenSecretManagerMetrics from being registered.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532413#comment-17532413 ] Hector Sandoval Chaverri commented on HADOOP-18167: --- [~ayushtkn] I haven't been able to repro by starting ResourceManager, but I see this can happen if there's a call to MetricsSystemImpl#init. I've created HADOOP-18222 to track this issue. > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10-3.patch, HADOOP-18167-branch-2.10-4.patch, > HADOOP-18167-branch-2.10.patch, HADOOP-18167-branch-3.3.patch > > Time Spent: 5h 50m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[ https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18167: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Add metrics to track delegation token secret manager operations > --- > > Key: HADOOP-18167 > URL: https://issues.apache.org/jira/browse/HADOOP-18167 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-18167-branch-2.10-2.patch, > HADOOP-18167-branch-2.10-3.patch, HADOOP-18167-branch-2.10-4.patch, > HADOOP-18167-branch-2.10.patch, HADOOP-18167-branch-3.3.patch > > Time Spent: 6h 10m > Remaining Estimate: 0h > > New metrics to track operations that store, update and remove delegation > tokens in implementations of AbstractDelegationTokenSecretManager. This will > help evaluate the impact of using different secret managers and add > optimizations. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18264) ZKDelegationTokenSecretManager should handle duplicate Token sequenceNums
Hector Sandoval Chaverri created HADOOP-18264: - Summary: ZKDelegationTokenSecretManager should handle duplicate Token sequenceNums Key: HADOOP-18264 URL: https://issues.apache.org/jira/browse/HADOOP-18264 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Hector Sandoval Chaverri The ZKDelegationTokenSecretManager relies on the TokenIdentifier sequenceNumber to identify each Token in the ZK Store. It's possible for multiple TokenIdentifiers to share the same sequenceNumber, as this is an int that can overflow. The AbstractDelegationTokenSecretManager uses a Map so all properties in the TokenIdentifier must match. ZKDelegationTokenSecretManager should follow the same logic. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18264) ZKDelegationTokenSecretManager should handle duplicate Token sequenceNums
[ https://issues.apache.org/jira/browse/HADOOP-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18264: -- Priority: Minor (was: Major) > ZKDelegationTokenSecretManager should handle duplicate Token sequenceNums > - > > Key: HADOOP-18264 > URL: https://issues.apache.org/jira/browse/HADOOP-18264 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Hector Sandoval Chaverri >Priority: Minor > > The ZKDelegationTokenSecretManager relies on the TokenIdentifier > sequenceNumber to identify each Token in the ZK Store. It's possible for > multiple TokenIdentifiers to share the same sequenceNumber, as this is an int > that can overflow. > The AbstractDelegationTokenSecretManager uses a Map DelegationTokenInformation> so all properties in the TokenIdentifier must > match. ZKDelegationTokenSecretManager should follow the same logic. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18535) Implement token storage solution based on MySQL
Hector Sandoval Chaverri created HADOOP-18535: - Summary: Implement token storage solution based on MySQL Key: HADOOP-18535 URL: https://issues.apache.org/jira/browse/HADOOP-18535 Project: Hadoop Common Issue Type: Improvement Reporter: Hector Sandoval Chaverri Assignee: Hector Sandoval Chaverri Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper. During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation will be made available, based on MySQL, to handle a higher number of writes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18535: -- Description: Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper. During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation has been created, based on MySQL, to handle a higher number of writes. We measured the throughput of each token operation (create/renew/cancel) on different setups and obtained the following results: # Sending requests directly to Namenode (no RBF): Token creations: 290 reqs per sec Token renewals: 86 reqs per sec Token cancellations: 97 reqs per sec # Sending requests to routers using Zookeeper based secret manager: Token creations: 31 reqs per sec Token renewals: 29 reqs per sec Token cancellations: 40 reqs per sec # Sending requests to routers using SQL based secret manager: Token creations: 241 reqs per sec Token renewals: 103 reqs per sec Token cancellations: 114 reqs per sec We noticed a significant improvement when using a SQL secret manager, comparable to the throughput offered by Namenodes. For this reason, was: Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper. During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation will be made available, based on MySQL, to handle a higher number of writes. > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. For this reason, -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18535: -- Description: Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper. During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation has been created, based on MySQL, to handle a higher number of writes. We measured the throughput of each token operation (create/renew/cancel) on different setups and obtained the following results: # Sending requests directly to Namenode (no RBF): Token creations: 290 reqs per sec Token renewals: 86 reqs per sec Token cancellations: 97 reqs per sec # Sending requests to routers using Zookeeper based secret manager: Token creations: 31 reqs per sec Token renewals: 29 reqs per sec Token cancellations: 40 reqs per sec # Sending requests to routers using SQL based secret manager: Token creations: 241 reqs per sec Token renewals: 103 reqs per sec Token cancellations: 114 reqs per sec We noticed a significant improvement when using a SQL secret manager, comparable to the throughput offered by Namenodes. was: Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper. During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation has been created, based on MySQL, to handle a higher number of writes. We measured the throughput of each token operation (create/renew/cancel) on different setups and obtained the following results: # Sending requests directly to Namenode (no RBF): Token creations: 290 reqs per sec Token renewals: 86 reqs per sec Token cancellations: 97 reqs per sec # Sending requests to routers using Zookeeper based secret manager: Token creations: 31 reqs per sec Token renewals: 29 reqs per sec Token cancellations: 40 reqs per sec # Sending requests to routers using SQL based secret manager: Token creations: 241 reqs per sec Token renewals: 103 reqs per sec Token cancellations: 114 reqs per sec We noticed a significant improvement when using a SQL secret manager, comparable to the throughput offered by Namenodes. For this reason, > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
