Re: Review Request 55482: AM sizing for LLAP - modify YARN min container size to a smaller value leaving all other properties as is

2017-01-13 Thread Siddharth Seth

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55482/#review161628
---




ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
(line 127)
<https://reviews.apache.org/r/55482/#comment232890>

This value gets changed based on the new ramPerContainer calculation, but 
somehow does not show up in recommendations.

Needs to be de-linked from the rounding, and stick to the old value.

I think the "minContainerSize" = user suggested value, as well as the 
rounding done for ramPerContainer will affect this.



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
(line 1044)
<https://reviews.apache.org/r/55482/#comment232887>

If we leave minContainerSize unchanged here - the old computations continue 
as is, correct?
So there'll be no change in containers, or ramPerContainer.

Finally when ramPerContainer is normalized against the new minContainerSize 
- a new value will be set.



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
(line 1069)
<https://reviews.apache.org/r/55482/#comment232888>

Tnis should be a roundup instead of a roundDown.

int(ceil(float(cluster["ramPerContainer"]) / cluster["minContainerRam"]) * 
cluster["minContainerRam"])



ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
<https://reviews.apache.org/r/55482/#comment232886>

Think we should retain this rounding functionality, before rounding to the 
set minContainerSize. That can affect whether we go above or below the 
minContainerSize.


- Siddharth Seth


On Jan. 13, 2017, 1:41 a.m., Sumit Mohanty wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55482/
> ---
> 
> (Updated Jan. 13, 2017, 1:41 a.m.)
> 
> 
> Review request for Ambari, Siddharth Seth, Swapan Shridhar, and Sid Wagle.
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A large value for YARN min container size reduces the number of AMs that can 
> be created for llap.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/scripts/stack_advisor.py abfab87 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 480e73d 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 
> cf7134e 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> d781147 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> 02be812 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 378522e 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> a7ccdfe 
> 
> Diff: https://reviews.apache.org/r/55482/diff/
> 
> 
> Testing
> ---
> 
> Ran all stack advisor tests manually.
> 
> --
> Ran 270 tests in 6.771s
> 
> OK
> --
> Total run:1151
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Sumit Mohanty
> 
>



Re: Review Request 55482: AM sizing for LLAP - modify YARN min container size to a smaller value leaving all other properties as is

2017-01-13 Thread Siddharth Seth


> On Jan. 13, 2017, 10:06 p.m., Siddharth Seth wrote:
> > Looked at the changes in stack-advisor. Mostly look good to me.
> > Was a little confused by the change to minContainerSize based on whether 
> > the used change the yarn min-allocation-mb property.
> 
> Sumit Mohanty wrote:
> So the scenarios that I am trying to capture are:
> * Initial cluster creation - SA will ignore what value it gets in 
> yarn-site and use the computation strategy it has
> * Validation call, call when user lands on an UI page, call when any 
> other config changes - SA will use whatever value that comes through 
> "yarn-site" (this is by definition considered an user provided value)
> * If user change the min container size then it will be indicated in the 
> changed_configuration[] section and then the user supplied new value is used.

Sorry. I missed some stuff. Ran into some problems while running this on a 
cluster. Looking some more. Will update comments after that.
e.g. Setting min-container-size to 2048, reduced the map size to 2048 instead 
of the old value of 3072.


- Siddharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55482/#review161592
---


On Jan. 13, 2017, 1:41 a.m., Sumit Mohanty wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55482/
> ---
> 
> (Updated Jan. 13, 2017, 1:41 a.m.)
> 
> 
> Review request for Ambari, Siddharth Seth, Swapan Shridhar, and Sid Wagle.
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A large value for YARN min container size reduces the number of AMs that can 
> be created for llap.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/scripts/stack_advisor.py abfab87 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 480e73d 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 
> cf7134e 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> d781147 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> 02be812 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 378522e 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> a7ccdfe 
> 
> Diff: https://reviews.apache.org/r/55482/diff/
> 
> 
> Testing
> ---
> 
> Ran all stack advisor tests manually.
> 
> --
> Ran 270 tests in 6.771s
> 
> OK
> --
> Total run:1151
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Sumit Mohanty
> 
>



Re: Review Request 55482: AM sizing for LLAP - modify YARN min container size to a smaller value leaving all other properties as is

2017-01-13 Thread Siddharth Seth

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55482/#review161592
---



Looked at the changes in stack-advisor. Mostly look good to me.
Was a little confused by the change to minContainerSize based on whether the 
used change the yarn min-allocation-mb property.

- Siddharth Seth


On Jan. 13, 2017, 1:41 a.m., Sumit Mohanty wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55482/
> ---
> 
> (Updated Jan. 13, 2017, 1:41 a.m.)
> 
> 
> Review request for Ambari, Siddharth Seth, Swapan Shridhar, and Sid Wagle.
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> A large value for YARN min container size reduces the number of AMs that can 
> be created for llap.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/scripts/stack_advisor.py abfab87 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 480e73d 
>   ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 
> cf7134e 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> d781147 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> 02be812 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 378522e 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> a7ccdfe 
> 
> Diff: https://reviews.apache.org/r/55482/diff/
> 
> 
> Testing
> ---
> 
> Ran all stack advisor tests manually.
> 
> --
> Ran 270 tests in 6.771s
> 
> OK
> --
> Total run:1151
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Sumit Mohanty
> 
>



Re: Review Request 53801: AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config calculations.

2016-11-18 Thread Siddharth Seth

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53801/#review156328
---


Ship it!




+1 for the stack_advisor computations.

- Siddharth Seth


On Nov. 18, 2016, 2:34 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53801/
> ---
> 
> (Updated Nov. 18, 2016, 2:34 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Madhuvanthi Radhakrishnan, 
> Sumit Mohanty, and Siddharth Seth.
> 
> 
> Bugs: AMBARI-18901
> https://issues.apache.org/jira/browse/AMBARI-18901
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP 
> config calculations.
> 
> Below is the calculation logic used:
> 
> 
> **
> **
> 
> 
> ---
> For use with default setup - Ambari managed queue
> 
> Parameteres
> numRequestedLlapNodes UserInput
> tezAmContainerSizeComputed
> memoryPerThread   Computed// Set as parameter in 
> HiveConf. user can override in Advanced
> numConcurrentQueries  Computed// user can override in Advanced
> maxConcurrentQueries  Computed// Max value for Slider
> numLlapNodes  Computed// Can be lower for 
> small clusters. user can override in Advanced
> sliderAmContainerSize Computed// user can override in Advanced
> numExecutorsPerNode   Computed// user can override in Advanced  | 
> TODO: This and memPerThread are about the same when taking daemonSize, and 
> cache into considertaion
> cacheMemoryPerNodeComputed// user can override in 
> Advanced
> amFraction1   // Set 
> to 1 for Ambari controlled queued. | TODO Factor into concurrency for user 
> provided queues
> llapQueueFraction Computed// Computed by Ambari. 
> (Avoid changing if the current value is > computed value, and the user 
> specified the current value?)
> 
> 
> numClusterNodes   ClusterParameter
> nmMemoryPerNode   ClusterParameter
> nmCpusPerNode ClusterParameter
> minContainerSize  ClusterParameter
> 
> 
> CONSTANT DEFAULT_EXECUTOR_TO_AM_RATIO = 20;
> CONSTANT MIN_EXECUTOR_TO_AM_RATIO = 10;
> CONSTANT MAX_CONCURRENT_QUERIES = 32;
> 
> nmMemoryPerNodeNormalized = normalizeDown(nmMemoryPerNode, minContainerSize);
> 
> totalClusterCapacity = numClusterNodes * nmMemoryPerNodeNormalized;
> totalllapMemory = numRequestedLlapNodes * nmMemoryPerNodeNormalized;
> amCapacityAvailable = totalLlapMemory; // For the LLAP queue - the AM 
> fraction is set to 1
> 
> sliderAmSize -> Current calculations remain unchanged. (<=, < fixes)
> 
> llapMemoryTezAMsAndDaemons = totalllapMemory - sliderAmSize;
> FAIL("Not enough capacity available on the cluster to run LLAP") if 
> (llapMemoryTezAMsAndDaemons < 2 * minContainerSize);
> 
> tezAmContainerSize = (totalClusterCapacity, minContainerSize) {
>   desiredSize = { // This part is unchanged from current calculations.
>   if (totalClusterCapacity <= 4096) {
>   return 256;
>   } else if (totalClusterCapacity <= 73728) {
>   return 512;
>   } else {
>   return 1536;
>   }
>   }
>   return normalizeUp(desiredSize, minContainerSize);
> }
> 
> memoryPerThread = (nmMemoryPerNodeNormalized, nmCpusPerNode) { // TODO: Not 
> linear. e.g. 1024 to 1025 goes from 2 executors to 1.
>   if (userSpecifiedValue) {
>   return userSpecifiedValue;
>   } else if (nmMemoryPerNodeNormalized <= 1024) {
>   return Math.min(512, nmMemoryPerNodeNormalized)
>   } else if (nmMemoryPerNodeNormalized <= 4096 ) {
>   return 1024;
>   } else if (nmMemoryPerNodeNormalized <= 10240) {
>   return 2048;
>   } else if (nmMemoryPerNodeNormalized <=

Re: Review Request 51104: AMBARI-18150. Add Hive2/HiveServer2 config "hive.metastore.event.listeners" as empty value in hive-interactive-site.

2016-08-15 Thread Siddharth Seth

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51104/#review145795
---


Ship it!




Ship It!

- Siddharth Seth


On Aug. 15, 2016, 8:20 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51104/
> ---
> 
> (Updated Aug. 15, 2016, 8:20 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Sumit Mohanty, Siddharth 
> Seth, and Thejas Nair.
> 
> 
> Bugs: AMBARI-18150
> https://issues.apache.org/jira/browse/AMBARI-18150
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - This is done because this is a custom config which if added to Hive1 will 
> get automatically inherited to Hive2. We don't want that to happen. 
> - The inheritance of config from Hive1 to Hive2 can be stopped by adding it a 
> empty config in Hive2.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
>  14ab470 
> 
> Diff: https://reviews.apache.org/r/51104/diff/
> 
> 
> Testing
> ---
> 
> Deployment:
> - Tested the deployment on new cluster.
> 
> UT: 
> - - Python UT passes.
> - Jenkins pending.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>



Re: Review Request 49014: AMBARI-17331. Determine Tez for Hive2 config 'tez.am.resource.memory.mb' based on cluster capacity.

2016-06-21 Thread Siddharth Seth

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49014/#review138924
---


Ship it!




Looks good. The Xmx change isn't required since it's not set for tez either 
(and tez will set this to 80% of the container size in this case)

- Siddharth Seth


On June 21, 2016, 7:59 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49014/
> ---
> 
> (Updated June 21, 2016, 7:59 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-17331
> https://issues.apache.org/jira/browse/AMBARI-17331
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The current computations done for tez ends up with a large AM size.
> For LLAP, we'd like stack-advisor to change these computations.
> 
> - Total Cluster Memory <=4GB - Tez AM size = 256MB and then normalized based 
> on YARN minimum container size.
> - Total Cluster Memory >4GB && <= 72GB - Tez AM size = 512MB and then 
> normalized based on YARN minimum container size.
> - Total Cluster Memory >72GB - Tez AM size = 1536MB and then normalized based 
> on YARN minimum container size.
> 
> 
> Fix for safeguaring 'llap' Queue creation to max. size of 100%, in case the 
> minimum size required caluclated size is more than 100%.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
>  3c83c5c 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 76654c3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 1bc53ea 
> 
> Diff: https://reviews.apache.org/r/49014/diff/
> 
> 
> Testing
> ---
> 
> Python UT added.
>   - Python UT passes.
> 
> 
> Thanks,
> 
> Swapan Shridhar
> 
>