Re: Review Request 55482: AM sizing for LLAP - modify YARN min container size to a smaller value leaving all other properties as is
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55482/#review161628 --- ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 127) <https://reviews.apache.org/r/55482/#comment232890> This value gets changed based on the new ramPerContainer calculation, but somehow does not show up in recommendations. Needs to be de-linked from the rounding, and stick to the old value. I think the "minContainerSize" = user suggested value, as well as the rounding done for ramPerContainer will affect this. ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 1044) <https://reviews.apache.org/r/55482/#comment232887> If we leave minContainerSize unchanged here - the old computations continue as is, correct? So there'll be no change in containers, or ramPerContainer. Finally when ramPerContainer is normalized against the new minContainerSize - a new value will be set. ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py (line 1069) <https://reviews.apache.org/r/55482/#comment232888> Tnis should be a roundup instead of a roundDown. int(ceil(float(cluster["ramPerContainer"]) / cluster["minContainerRam"]) * cluster["minContainerRam"]) ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py <https://reviews.apache.org/r/55482/#comment232886> Think we should retain this rounding functionality, before rounding to the set minContainerSize. That can affect whether we go above or below the minContainerSize. - Siddharth Seth On Jan. 13, 2017, 1:41 a.m., Sumit Mohanty wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55482/ > --- > > (Updated Jan. 13, 2017, 1:41 a.m.) > > > Review request for Ambari, Siddharth Seth, Swapan Shridhar, and Sid Wagle. > > > Repository: ambari > > > Description > --- > > A large value for YARN min container size reduces the number of AMs that can > be created for llap. > > > Diffs > - > > ambari-server/src/main/resources/scripts/stack_advisor.py abfab87 > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 480e73d > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py > cf7134e > ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py > d781147 > ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py > 02be812 > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 378522e > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > a7ccdfe > > Diff: https://reviews.apache.org/r/55482/diff/ > > > Testing > --- > > Ran all stack advisor tests manually. > > -- > Ran 270 tests in 6.771s > > OK > -- > Total run:1151 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Sumit Mohanty > >
Re: Review Request 55482: AM sizing for LLAP - modify YARN min container size to a smaller value leaving all other properties as is
> On Jan. 13, 2017, 10:06 p.m., Siddharth Seth wrote: > > Looked at the changes in stack-advisor. Mostly look good to me. > > Was a little confused by the change to minContainerSize based on whether > > the used change the yarn min-allocation-mb property. > > Sumit Mohanty wrote: > So the scenarios that I am trying to capture are: > * Initial cluster creation - SA will ignore what value it gets in > yarn-site and use the computation strategy it has > * Validation call, call when user lands on an UI page, call when any > other config changes - SA will use whatever value that comes through > "yarn-site" (this is by definition considered an user provided value) > * If user change the min container size then it will be indicated in the > changed_configuration[] section and then the user supplied new value is used. Sorry. I missed some stuff. Ran into some problems while running this on a cluster. Looking some more. Will update comments after that. e.g. Setting min-container-size to 2048, reduced the map size to 2048 instead of the old value of 3072. - Siddharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55482/#review161592 --- On Jan. 13, 2017, 1:41 a.m., Sumit Mohanty wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55482/ > --- > > (Updated Jan. 13, 2017, 1:41 a.m.) > > > Review request for Ambari, Siddharth Seth, Swapan Shridhar, and Sid Wagle. > > > Repository: ambari > > > Description > --- > > A large value for YARN min container size reduces the number of AMs that can > be created for llap. > > > Diffs > - > > ambari-server/src/main/resources/scripts/stack_advisor.py abfab87 > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 480e73d > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py > cf7134e > ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py > d781147 > ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py > 02be812 > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 378522e > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > a7ccdfe > > Diff: https://reviews.apache.org/r/55482/diff/ > > > Testing > --- > > Ran all stack advisor tests manually. > > -- > Ran 270 tests in 6.771s > > OK > -- > Total run:1151 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Sumit Mohanty > >
Re: Review Request 55482: AM sizing for LLAP - modify YARN min container size to a smaller value leaving all other properties as is
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55482/#review161592 --- Looked at the changes in stack-advisor. Mostly look good to me. Was a little confused by the change to minContainerSize based on whether the used change the yarn min-allocation-mb property. - Siddharth Seth On Jan. 13, 2017, 1:41 a.m., Sumit Mohanty wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55482/ > --- > > (Updated Jan. 13, 2017, 1:41 a.m.) > > > Review request for Ambari, Siddharth Seth, Swapan Shridhar, and Sid Wagle. > > > Repository: ambari > > > Description > --- > > A large value for YARN min container size reduces the number of AMs that can > be created for llap. > > > Diffs > - > > ambari-server/src/main/resources/scripts/stack_advisor.py abfab87 > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py > 480e73d > ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py > cf7134e > ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py > d781147 > ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py > 02be812 > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py > 378522e > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > a7ccdfe > > Diff: https://reviews.apache.org/r/55482/diff/ > > > Testing > --- > > Ran all stack advisor tests manually. > > -- > Ran 270 tests in 6.771s > > OK > -- > Total run:1151 > Total errors:0 > Total failures:0 > OK > > > Thanks, > > Sumit Mohanty > >
Re: Review Request 53801: AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config calculations.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53801/#review156328 --- Ship it! +1 for the stack_advisor computations. - Siddharth Seth On Nov. 18, 2016, 2:34 a.m., Swapan Shridhar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53801/ > --- > > (Updated Nov. 18, 2016, 2:34 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Madhuvanthi Radhakrishnan, > Sumit Mohanty, and Siddharth Seth. > > > Bugs: AMBARI-18901 > https://issues.apache.org/jira/browse/AMBARI-18901 > > > Repository: ambari > > > Description > --- > > AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP > config calculations. > > Below is the calculation logic used: > > > ** > ** > > > --- > For use with default setup - Ambari managed queue > > Parameteres > numRequestedLlapNodes UserInput > tezAmContainerSizeComputed > memoryPerThread Computed// Set as parameter in > HiveConf. user can override in Advanced > numConcurrentQueries Computed// user can override in Advanced > maxConcurrentQueries Computed// Max value for Slider > numLlapNodes Computed// Can be lower for > small clusters. user can override in Advanced > sliderAmContainerSize Computed// user can override in Advanced > numExecutorsPerNode Computed// user can override in Advanced | > TODO: This and memPerThread are about the same when taking daemonSize, and > cache into considertaion > cacheMemoryPerNodeComputed// user can override in > Advanced > amFraction1 // Set > to 1 for Ambari controlled queued. | TODO Factor into concurrency for user > provided queues > llapQueueFraction Computed// Computed by Ambari. > (Avoid changing if the current value is > computed value, and the user > specified the current value?) > > > numClusterNodes ClusterParameter > nmMemoryPerNode ClusterParameter > nmCpusPerNode ClusterParameter > minContainerSize ClusterParameter > > > CONSTANT DEFAULT_EXECUTOR_TO_AM_RATIO = 20; > CONSTANT MIN_EXECUTOR_TO_AM_RATIO = 10; > CONSTANT MAX_CONCURRENT_QUERIES = 32; > > nmMemoryPerNodeNormalized = normalizeDown(nmMemoryPerNode, minContainerSize); > > totalClusterCapacity = numClusterNodes * nmMemoryPerNodeNormalized; > totalllapMemory = numRequestedLlapNodes * nmMemoryPerNodeNormalized; > amCapacityAvailable = totalLlapMemory; // For the LLAP queue - the AM > fraction is set to 1 > > sliderAmSize -> Current calculations remain unchanged. (<=, < fixes) > > llapMemoryTezAMsAndDaemons = totalllapMemory - sliderAmSize; > FAIL("Not enough capacity available on the cluster to run LLAP") if > (llapMemoryTezAMsAndDaemons < 2 * minContainerSize); > > tezAmContainerSize = (totalClusterCapacity, minContainerSize) { > desiredSize = { // This part is unchanged from current calculations. > if (totalClusterCapacity <= 4096) { > return 256; > } else if (totalClusterCapacity <= 73728) { > return 512; > } else { > return 1536; > } > } > return normalizeUp(desiredSize, minContainerSize); > } > > memoryPerThread = (nmMemoryPerNodeNormalized, nmCpusPerNode) { // TODO: Not > linear. e.g. 1024 to 1025 goes from 2 executors to 1. > if (userSpecifiedValue) { > return userSpecifiedValue; > } else if (nmMemoryPerNodeNormalized <= 1024) { > return Math.min(512, nmMemoryPerNodeNormalized) > } else if (nmMemoryPerNodeNormalized <= 4096 ) { > return 1024; > } else if (nmMemoryPerNodeNormalized <= 10240) { > return 2048; > } else if (nmMemoryPerNodeNormalized <=
Re: Review Request 51104: AMBARI-18150. Add Hive2/HiveServer2 config "hive.metastore.event.listeners" as empty value in hive-interactive-site.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51104/#review145795 --- Ship it! Ship It! - Siddharth Seth On Aug. 15, 2016, 8:20 p.m., Swapan Shridhar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51104/ > --- > > (Updated Aug. 15, 2016, 8:20 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Sumit Mohanty, Siddharth > Seth, and Thejas Nair. > > > Bugs: AMBARI-18150 > https://issues.apache.org/jira/browse/AMBARI-18150 > > > Repository: ambari > > > Description > --- > > - This is done because this is a custom config which if added to Hive1 will > get automatically inherited to Hive2. We don't want that to happen. > - The inheritance of config from Hive1 to Hive2 can be stopped by adding it a > empty config in Hive2. > > > Diffs > - > > > ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml > 14ab470 > > Diff: https://reviews.apache.org/r/51104/diff/ > > > Testing > --- > > Deployment: > - Tested the deployment on new cluster. > > UT: > - - Python UT passes. > - Jenkins pending. > > > Thanks, > > Swapan Shridhar > >
Re: Review Request 49014: AMBARI-17331. Determine Tez for Hive2 config 'tez.am.resource.memory.mb' based on cluster capacity.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49014/#review138924 --- Ship it! Looks good. The Xmx change isn't required since it's not set for tez either (and tez will set this to 80% of the container size in this case) - Siddharth Seth On June 21, 2016, 7:59 p.m., Swapan Shridhar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49014/ > --- > > (Updated June 21, 2016, 7:59 p.m.) > > > Review request for Ambari, Alejandro Fernandez and Sumit Mohanty. > > > Bugs: AMBARI-17331 > https://issues.apache.org/jira/browse/AMBARI-17331 > > > Repository: ambari > > > Description > --- > > The current computations done for tez ends up with a large AM size. > For LLAP, we'd like stack-advisor to change these computations. > > - Total Cluster Memory <=4GB - Tez AM size = 256MB and then normalized based > on YARN minimum container size. > - Total Cluster Memory >4GB && <= 72GB - Tez AM size = 512MB and then > normalized based on YARN minimum container size. > - Total Cluster Memory >72GB - Tez AM size = 1536MB and then normalized based > on YARN minimum container size. > > > Fix for safeguaring 'llap' Queue creation to max. size of 100%, in case the > minimum size required caluclated size is more than 100%. > > > Diffs > - > > > ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml > 3c83c5c > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 76654c3 > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > 1bc53ea > > Diff: https://reviews.apache.org/r/49014/diff/ > > > Testing > --- > > Python UT added. > - Python UT passes. > > > Thanks, > > Swapan Shridhar > >