[
https://issues.apache.org/jira/browse/AMBARI-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Mitic updated AMBARI-8188:
-------------------------------
Description:
This is an umbrella Jira that tracks improvements in the Ambari cluster
deployment performance, mainly targeting cloud environments. What is special
about cloud environments is that VM images could have preinstalled RPM/DEB
packages.
Based on experiments run on public Azure on Large VMs SKU, Azure Storage (WASB)
as a default File System, and HDP 2.1 stack:
Ambari cluster deployment took:
* Install services stage – ~1.5 minute in average
** Given that all components are already preinstalled during this step Ambari
just replays all install scripts and skips over the actual RPM/DEB package
installations.
* Start services stage – ~12 minutes in average
** During this step Ambari performs additional setup of components/services
required before the services can be started up. Example of such setup steps are
Namenode format, Oozie and Hive metastore setup, Oozie and WebHCat shared lib
setup.
Detailed breakdown of the service start stage:
Disclaimer: Numbers have changed with HDP 2.2, will update the Jira separately.
* NameNode format and exit safemode – 1:15
* History Server state setup (permissions and ownership FileSystem operations)
– 0:50
* Hive Metastore setup – 0:30
* Hive Server 2 state setup (permissions and ownership FileSystem operations) –
1:20
* Oozie server metastore and shared lib setup (unzip + copy many small blobs +
other FS operations) – 3:25
* WebHCat shared lib setup – 1:30
* Overall services startup and scheduling overhead – 2:00
was:
This is an umbrella Jira that tracks improvements in the Ambari cluster
deployment performance, mainly targeting cloud environments. What is special
about cloud environments is that VM images could have preinstalled RPM/DEB
packages.
Based on experiments run on public Azure on Large VMs SKU, Azure Storage (WASB)
as a default File System, and HDP 2.1 stack:
Ambari cluster deployment took:
* Install services stage – ~1.5 minute in average
** Given that all components are already preinstalled during this step Ambari
just replays all install scripts and skips over the actual RPM/DEB package
installations.
* Start services stage – ~12 minutes in average
** During this step Ambari performs additional setup of components/services
required before the services can be started up. Example of such setup steps are
Namenode format, Oozie and Hive metastore setup, Oozie and WebHCat shared lib
setup.
Detailed breakdown of the service start stage:
* NameNode format and exit safemode – 1:15
* History Server state setup (permissions and ownership FileSystem operations)
– 0:50
* Hive Metastore setup – 0:30
* Hive Server 2 state setup (permissions and ownership FileSystem operations) –
1:20
* Oozie server metastore and shared lib setup (unzip + copy many small blobs +
other FS operations) – 3:25
* WebHCat shared lib setup – 1:30
* Overall services startup and scheduling overhead – 2:00
> Improve Ambari deployment performance in cloud environments
> -----------------------------------------------------------
>
> Key: AMBARI-8188
> URL: https://issues.apache.org/jira/browse/AMBARI-8188
> Project: Ambari
> Issue Type: Improvement
> Affects Versions: 2.0.0
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
>
> This is an umbrella Jira that tracks improvements in the Ambari cluster
> deployment performance, mainly targeting cloud environments. What is special
> about cloud environments is that VM images could have preinstalled RPM/DEB
> packages.
> Based on experiments run on public Azure on Large VMs SKU, Azure Storage
> (WASB) as a default File System, and HDP 2.1 stack:
> Ambari cluster deployment took:
> * Install services stage – ~1.5 minute in average
> ** Given that all components are already preinstalled during this step Ambari
> just replays all install scripts and skips over the actual RPM/DEB package
> installations.
> * Start services stage – ~12 minutes in average
> ** During this step Ambari performs additional setup of components/services
> required before the services can be started up. Example of such setup steps
> are Namenode format, Oozie and Hive metastore setup, Oozie and WebHCat shared
> lib setup.
> Detailed breakdown of the service start stage:
> Disclaimer: Numbers have changed with HDP 2.2, will update the Jira
> separately.
> * NameNode format and exit safemode – 1:15
> * History Server state setup (permissions and ownership FileSystem
> operations) – 0:50
> * Hive Metastore setup – 0:30
> * Hive Server 2 state setup (permissions and ownership FileSystem operations)
> – 1:20
> * Oozie server metastore and shared lib setup (unzip + copy many small blobs
> + other FS operations) – 3:25
> * WebHCat shared lib setup – 1:30
> * Overall services startup and scheduling overhead – 2:00
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
