[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18535: Hadoop Flags: Reviewed Target Version/s: 3.3.6, 3.4.0 > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0, 3.3.6 >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18535: Component/s: common > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 3.4.0, 3.3.6 >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18535: Affects Version/s: 3.3.6 3.4.0 > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.4.0, 3.3.6 >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18535: Fix Version/s: 3.4.0 > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.6 > > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated HADOOP-18535: Fix Version/s: (was: 3.4.0) > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > Fix For: 3.3.6 > > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18535: -- Description: Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper. During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation has been created, based on MySQL, to handle a higher number of writes. We measured the throughput of each token operation (create/renew/cancel) on different setups and obtained the following results: # Sending requests directly to Namenode (no RBF): Token creations: 290 reqs per sec Token renewals: 86 reqs per sec Token cancellations: 97 reqs per sec # Sending requests to routers using Zookeeper based secret manager: Token creations: 31 reqs per sec Token renewals: 29 reqs per sec Token cancellations: 40 reqs per sec # Sending requests to routers using SQL based secret manager: Token creations: 241 reqs per sec Token renewals: 103 reqs per sec Token cancellations: 114 reqs per sec We noticed a significant improvement when using a SQL secret manager, comparable to the throughput offered by Namenodes. was: Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper. During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation has been created, based on MySQL, to handle a higher number of writes. We measured the throughput of each token operation (create/renew/cancel) on different setups and obtained the following results: # Sending requests directly to Namenode (no RBF): Token creations: 290 reqs per sec Token renewals: 86 reqs per sec Token cancellations: 97 reqs per sec # Sending requests to routers using Zookeeper based secret manager: Token creations: 31 reqs per sec Token renewals: 29 reqs per sec Token cancellations: 40 reqs per sec # Sending requests to routers using SQL based secret manager: Token creations: 241 reqs per sec Token renewals: 103 reqs per sec Token cancellations: 114 reqs per sec We noticed a significant improvement when using a SQL secret manager, comparable to the throughput offered by Namenodes. For this reason, > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hector Sandoval Chaverri updated HADOOP-18535: -- Description: Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper. During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation has been created, based on MySQL, to handle a higher number of writes. We measured the throughput of each token operation (create/renew/cancel) on different setups and obtained the following results: # Sending requests directly to Namenode (no RBF): Token creations: 290 reqs per sec Token renewals: 86 reqs per sec Token cancellations: 97 reqs per sec # Sending requests to routers using Zookeeper based secret manager: Token creations: 31 reqs per sec Token renewals: 29 reqs per sec Token cancellations: 40 reqs per sec # Sending requests to routers using SQL based secret manager: Token creations: 241 reqs per sec Token renewals: 103 reqs per sec Token cancellations: 114 reqs per sec We noticed a significant improvement when using a SQL secret manager, comparable to the throughput offered by Namenodes. For this reason, was: Hadoop RBF supports custom implementations of secret managers. At the moment, the only available implementation is ZKDelegationTokenSecretManagerImpl, which stores tokens and delegation keys in Zookeeper. During our investigation, we found that the performance of routers is limited by the writes to the Zookeeper token store, which impacts requests for token creation, renewal and cancellation. An alternative secret manager implementation will be made available, based on MySQL, to handle a higher number of writes. > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation has been created, based on MySQL, to handle a higher number of > writes. > We measured the throughput of each token operation (create/renew/cancel) on > different setups and obtained the following results: > # Sending requests directly to Namenode (no RBF): > Token creations: 290 reqs per sec > Token renewals: 86 reqs per sec > Token cancellations: 97 reqs per sec > # Sending requests to routers using Zookeeper based secret manager: > Token creations: 31 reqs per sec > Token renewals: 29 reqs per sec > Token cancellations: 40 reqs per sec > # Sending requests to routers using SQL based secret manager: > Token creations: 241 reqs per sec > Token renewals: 103 reqs per sec > Token cancellations: 114 reqs per sec > We noticed a significant improvement when using a SQL secret manager, > comparable to the throughput offered by Namenodes. For this reason, -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18535) Implement token storage solution based on MySQL
[ https://issues.apache.org/jira/browse/HADOOP-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18535: Labels: pull-request-available (was: ) > Implement token storage solution based on MySQL > --- > > Key: HADOOP-18535 > URL: https://issues.apache.org/jira/browse/HADOOP-18535 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Hector Sandoval Chaverri >Assignee: Hector Sandoval Chaverri >Priority: Major > Labels: pull-request-available > > Hadoop RBF supports custom implementations of secret managers. At the moment, > the only available implementation is ZKDelegationTokenSecretManagerImpl, > which stores tokens and delegation keys in Zookeeper. > During our investigation, we found that the performance of routers is limited > by the writes to the Zookeeper token store, which impacts requests for token > creation, renewal and cancellation. An alternative secret manager > implementation will be made available, based on MySQL, to handle a higher > number of writes. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org