[jira] [Updated] (YARN-7399) Yarn services metadata storage improvement
[ https://issues.apache.org/jira/browse/YARN-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7399: Attachment: YARN-7399.png See the attached diagram for the current implementation and proposed refinement. This will reduce duplicated code for storing metadata, and support multiple storage type. > Yarn services metadata storage improvement > -- > > Key: YARN-7399 > URL: https://issues.apache.org/jira/browse/YARN-7399 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7399.png > > > In Slider, metadata is stored in user's home directory. Slider command line > interface interacts with HDFS directly to list deployed applications and > invoke YARN API or HDFS API to provide information to user. This design works > for a single user manage his/her own applications. When this design has been > ported to Yarn services, it becomes apparent that this design is difficult to > list all deployed applications on Hadoop cluster for administrator to manage > applications. Resource Manager needs to crawl through every user's home > directory to compile metadata about deployed applications. This can trigger > high load on namenode to list hundreds or thousands of list directory calls > owned by different users. Hence, it might be best to centralize the metadata > storage to Solr or HBase to reduce number of IO calls to namenode for manage > applications. > In Slider, one application is composed of metainfo, specifications in json, > and payload of zip file that contains application code and deployment code. > Both meta information, and zip file payload are stored in the same > application directory in HDFS. This works well for distributed applications > without central application manager that oversee all application. > In the next generation of application management, we like to centralize > metainfo and specifications in json to a centralized storage managed by YARN > user, and keep the payload zip file in user's home directory or in docker > registry. This arrangement can provide a faster lookup for metainfo when we > list all deployed applications and services on YARN dashboard. > When we centralize metainfo to YARN user, we also need to build ACL to > enforce who can manage applications, and make update. The current proposal is: > yarn.admin.acl - list of groups that can submit/reconfigure/pause/kill all > applications > normal users - submit/reconfigure/pause/kill his/her own applications -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7399) Yarn services metadata storage improvement
[ https://issues.apache.org/jira/browse/YARN-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7399: Attachment: YARN-7399.001.patch First draft of the storage API. Main features of this interface divided into two parts: # Ability to store template form of service configuration. The template can be instantiated by users. # Ability to store service configuration for a deployed instance of service. [~billie.rinaldi] [~gsaha] Please review if the interface covers all necessary operations. Thanks > Yarn services metadata storage improvement > -- > > Key: YARN-7399 > URL: https://issues.apache.org/jira/browse/YARN-7399 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7399.001.patch, YARN-7399.png > > > In Slider, metadata is stored in user's home directory. Slider command line > interface interacts with HDFS directly to list deployed applications and > invoke YARN API or HDFS API to provide information to user. This design works > for a single user manage his/her own applications. When this design has been > ported to Yarn services, it becomes apparent that this design is difficult to > list all deployed applications on Hadoop cluster for administrator to manage > applications. Resource Manager needs to crawl through every user's home > directory to compile metadata about deployed applications. This can trigger > high load on namenode to list hundreds or thousands of list directory calls > owned by different users. Hence, it might be best to centralize the metadata > storage to Solr or HBase to reduce number of IO calls to namenode for manage > applications. > In Slider, one application is composed of metainfo, specifications in json, > and payload of zip file that contains application code and deployment code. > Both meta information, and zip file payload are stored in the same > application directory in HDFS. This works well for distributed applications > without central application manager that oversee all application. > In the next generation of application management, we like to centralize > metainfo and specifications in json to a centralized storage managed by YARN > user, and keep the payload zip file in user's home directory or in docker > registry. This arrangement can provide a faster lookup for metainfo when we > list all deployed applications and services on YARN dashboard. > When we centralize metainfo to YARN user, we also need to build ACL to > enforce who can manage applications, and make update. The current proposal is: > yarn.admin.acl - list of groups that can submit/reconfigure/pause/kill all > applications > normal users - submit/reconfigure/pause/kill his/her own applications -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7399) Yarn services metadata storage improvement
[ https://issues.apache.org/jira/browse/YARN-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7399: Attachment: YARN-7399.002.patch [~billie.rinaldi], updated according to your suggestion. > Yarn services metadata storage improvement > -- > > Key: YARN-7399 > URL: https://issues.apache.org/jira/browse/YARN-7399 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-native-services >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7399.001.patch, YARN-7399.002.patch, YARN-7399.png > > > In Slider, metadata is stored in user's home directory. Slider command line > interface interacts with HDFS directly to list deployed applications and > invoke YARN API or HDFS API to provide information to user. This design works > for a single user manage his/her own applications. When this design has been > ported to Yarn services, it becomes apparent that this design is difficult to > list all deployed applications on Hadoop cluster for administrator to manage > applications. Resource Manager needs to crawl through every user's home > directory to compile metadata about deployed applications. This can trigger > high load on namenode to list hundreds or thousands of list directory calls > owned by different users. Hence, it might be best to centralize the metadata > storage to Solr or HBase to reduce number of IO calls to namenode for manage > applications. > In Slider, one application is composed of metainfo, specifications in json, > and payload of zip file that contains application code and deployment code. > Both meta information, and zip file payload are stored in the same > application directory in HDFS. This works well for distributed applications > without central application manager that oversee all application. > In the next generation of application management, we like to centralize > metainfo and specifications in json to a centralized storage managed by YARN > user, and keep the payload zip file in user's home directory or in docker > registry. This arrangement can provide a faster lookup for metainfo when we > list all deployed applications and services on YARN dashboard. > When we centralize metainfo to YARN user, we also need to build ACL to > enforce who can manage applications, and make update. The current proposal is: > yarn.admin.acl - list of groups that can submit/reconfigure/pause/kill all > applications > normal users - submit/reconfigure/pause/kill his/her own applications -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7399) Yarn services metadata storage improvement
[ https://issues.apache.org/jira/browse/YARN-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7399: Description: In Slider, metadata is stored in user's home directory. Slider command line interface interacts with HDFS directly to list deployed applications and invoke YARN API or HDFS API to provide information to user. This design works for a single user manage his/her own applications. When this design has been ported to Yarn services, it becomes apparent that this design is difficult to list all deployed applications on Hadoop cluster for administrator to manage applications. Resource Manager needs to crawl through every user's home directory to compile metadata about deployed applications. This can trigger high load on namenode to list hundreds or thousands of list directory calls owned by different users. Hence, it might be best to centralize the metadata storage to Solr or HBase to reduce number of IO calls to namenode for manage applications. In Slider, one application is composed of metainfo, specifications in json, and payload of zip file that contains application code and deployment code. Both meta information, and zip file payload are stored in the same application directory in HDFS. This works well for distributed applications without central application manager that oversee all application. In the next generation of application management, we like to centralize metainfo and specifications in json to a centralized storage managed by YARN user, and keep the payload zip file in user's home directory or in docker registry. This arrangement can provide a faster lookup for metainfo when we list all deployed applications and services on YARN dashboard. When we centralize metainfo to YARN user, we also need to build ACL to enforce who can manage applications, and make update. The current proposal is: yarn.admin.acl - list of groups that can submit/reconfigure/pause/kill all applications normal users - submit/reconfigure/pause/kill his/her own applications was:In Slider, metadata is stored in user's home directory. Slider command line interface interacts with HDFS directly to list deployed applications and invoke YARN API or HDFS API to provide information to user. This design works for a single user manage his/her own applications. When this design has been ported to Yarn services, it becomes apparent that this design is difficult to list all deployed applications on Hadoop cluster for administrator to manage applications. Resource Manager needs to crawl through every user's home directory to compile metadata about deployed applications. This can trigger high load on namenode to list hundreds or thousands of list directory calls owned by different users. Hence, it might be best to centralize the metadata storage to Solr or HBase to reduce number of IO calls to namenode for manage applications. > Yarn services metadata storage improvement > -- > > Key: YARN-7399 > URL: https://issues.apache.org/jira/browse/YARN-7399 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-native-services >Reporter: Eric Yang > > In Slider, metadata is stored in user's home directory. Slider command line > interface interacts with HDFS directly to list deployed applications and > invoke YARN API or HDFS API to provide information to user. This design works > for a single user manage his/her own applications. When this design has been > ported to Yarn services, it becomes apparent that this design is difficult to > list all deployed applications on Hadoop cluster for administrator to manage > applications. Resource Manager needs to crawl through every user's home > directory to compile metadata about deployed applications. This can trigger > high load on namenode to list hundreds or thousands of list directory calls > owned by different users. Hence, it might be best to centralize the metadata > storage to Solr or HBase to reduce number of IO calls to namenode for manage > applications. > In Slider, one application is composed of metainfo, specifications in json, > and payload of zip file that contains application code and deployment code. > Both meta information, and zip file payload are stored in the same > application directory in HDFS. This works well for distributed applications > without central application manager that oversee all application. > In the next generation of application management, we like to centralize > metainfo and specifications in json to a centralized storage managed by YARN > user, and keep the payload zip file in user's home directory or in docker > registry. This arrangement can provide a faster lookup for metainfo when we > list all deployed applications and services on YARN dashboard. > When we centralize metainfo to YARN
