Re: Review Request 47946: Yarn minimum container size calculation problem in stack advisor

[Daniel Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47946/
---

(Updated máj. 27, 2016, 10:14 de)


Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Oliver Szabo, 
Sandor Magyari, Sumit Mohanty, Swapan Shridhar, and Sebastian Toader.


Bugs: AMBARI-16921
https://issues.apache.org/jira/browse/AMBARI-16921


Repository: ambari


Description (updated)
---

Stack advisor gets yarn minimum container size property from only the 
"services" input, but does not take into account the already calculated 
configurations.
As a result memory size for hive interactive server is not correct, so it 
doesn't start.

Now services and configurations array are both considered. Does services even 
needed here?


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
8c5351f 

Diff: https://reviews.apache.org/r/47946/diff/


Testing
---

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 2:45.484s
[INFO] Finished at: Fri May 27 12:05:56 CEST 2016
[INFO] Final Memory: 119M/797M
[INFO] 


Thanks,

Daniel Gergely

Re: Review Request 47946: Yarn minimum container size calculation problem in stack advisor

[Swapan Shridhar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47946/#review135215
---



Good catch for requirement of looking into configurations also.


ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 
643)


Should we do max here, or just pick configurations if 
'yarn.scheduler.minimum-allocation-mb' value is there, as that's the latest 
value in terms of timeline compared to services. Your inputs please?


- Swapan Shridhar


On May 27, 2016, 10:14 a.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47946/
> ---
> 
> (Updated May 27, 2016, 10:14 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Oliver Szabo, 
> Sandor Magyari, Sumit Mohanty, Swapan Shridhar, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-16921
> https://issues.apache.org/jira/browse/AMBARI-16921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Stack advisor gets yarn minimum container size property from only the 
> "services" input, but does not take into account the already calculated 
> configurations.
> As a result memory size for hive interactive server is not correct, so it 
> doesn't start.
> 
> Now services and configurations array are both considered. Does services even 
> needed here?
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
> 
> Diff: https://reviews.apache.org/r/47946/diff/
> 
> 
> Testing
> ---
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:45.484s
> [INFO] Finished at: Fri May 27 12:05:56 CEST 2016
> [INFO] Final Memory: 119M/797M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Re: Review Request 47946: Yarn minimum container size calculation problem in stack advisor

[Daniel Gergely]

> On máj. 27, 2016, 11:05 de, Swapan Shridhar wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, 
> > line 643
> > 
> >
> > Should we do max here, or just pick configurations if 
> > 'yarn.scheduler.minimum-allocation-mb' value is there, as that's the latest 
> > value in terms of timeline compared to services. Your inputs please?

Basically I asked the same question. Services array might not needed at all, 
since the output of the property calculation (for yarn) is in the 
configurations array, so we can use it and that is the most accurate one. But 
services is used at many other places, so I did not want to remove it 
completely, because I might broke something.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47946/#review135215
---


On máj. 27, 2016, 10:14 de, Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47946/
> ---
> 
> (Updated máj. 27, 2016, 10:14 de)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Oliver Szabo, 
> Sandor Magyari, Sumit Mohanty, Swapan Shridhar, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-16921
> https://issues.apache.org/jira/browse/AMBARI-16921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Stack advisor gets yarn minimum container size property from only the 
> "services" input, but does not take into account the already calculated 
> configurations.
> As a result memory size for hive interactive server is not correct, so it 
> doesn't start.
> 
> Now services and configurations array are both considered. Does services even 
> needed here?
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
> 
> Diff: https://reviews.apache.org/r/47946/diff/
> 
> 
> Testing
> ---
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:45.484s
> [INFO] Finished at: Fri May 27 12:05:56 CEST 2016
> [INFO] Final Memory: 119M/797M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Re: Review Request 47946: Yarn minimum container size calculation problem in stack advisor

[Swapan Shridhar]

> On May 27, 2016, 11:05 a.m., Swapan Shridhar wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, 
> > line 643
> > 
> >
> > Should we do max here, or just pick configurations if 
> > 'yarn.scheduler.minimum-allocation-mb' value is there, as that's the latest 
> > value in terms of timeline compared to services. Your inputs please?
> 
> Daniel Gergely wrote:
> Basically I asked the same question. Services array might not needed at 
> all, since the output of the property calculation (for yarn) is in the 
> configurations array, so we can use it and that is the most accurate one. But 
> services is used at many other places, so I did not want to remove it 
> completely, because I might broke something.

Sure. How about the idea of look into configuration for a given config. If it 
exists, implies we changed/recommended it in current ST. Otherwise read from 
services where it should most probably be there. Otherwise raise.


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47946/#review135215
---


On May 27, 2016, 10:14 a.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47946/
> ---
> 
> (Updated May 27, 2016, 10:14 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Oliver Szabo, 
> Sandor Magyari, Sumit Mohanty, Swapan Shridhar, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-16921
> https://issues.apache.org/jira/browse/AMBARI-16921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Stack advisor gets yarn minimum container size property from only the 
> "services" input, but does not take into account the already calculated 
> configurations.
> As a result memory size for hive interactive server is not correct, so it 
> doesn't start.
> 
> Now services and configurations array are both considered. Does services even 
> needed here?
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
> 
> Diff: https://reviews.apache.org/r/47946/diff/
> 
> 
> Testing
> ---
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:45.484s
> [INFO] Finished at: Fri May 27 12:05:56 CEST 2016
> [INFO] Final Memory: 119M/797M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Re: Review Request 47946: Yarn minimum container size calculation problem in stack advisor

[Swapan Shridhar]

> On May 27, 2016, 11:05 a.m., Swapan Shridhar wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, 
> > line 643
> > 
> >
> > Should we do max here, or just pick configurations if 
> > 'yarn.scheduler.minimum-allocation-mb' value is there, as that's the latest 
> > value in terms of timeline compared to services. Your inputs please?
> 
> Daniel Gergely wrote:
> Basically I asked the same question. Services array might not needed at 
> all, since the output of the property calculation (for yarn) is in the 
> configurations array, so we can use it and that is the most accurate one. But 
> services is used at many other places, so I did not want to remove it 
> completely, because I might broke something.
> 
> Swapan Shridhar wrote:
> Sure. How about the idea of look into configuration for a given config. 
> If it exists, implies we changed/recommended it in current ST. Otherwise read 
> from services where it should most probably be there. Otherwise raise.

Does the above way sound foolproof?


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47946/#review135215
---


On May 27, 2016, 10:14 a.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/47946/
> ---
> 
> (Updated May 27, 2016, 10:14 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Oliver Szabo, 
> Sandor Magyari, Sumit Mohanty, Swapan Shridhar, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-16921
> https://issues.apache.org/jira/browse/AMBARI-16921
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Stack advisor gets yarn minimum container size property from only the 
> "services" input, but does not take into account the already calculated 
> configurations.
> As a result memory size for hive interactive server is not correct, so it 
> doesn't start.
> 
> Now services and configurations array are both considered. Does services even 
> needed here?
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 8c5351f 
> 
> Diff: https://reviews.apache.org/r/47946/diff/
> 
> 
> Testing
> ---
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 2:45.484s
> [INFO] Finished at: Fri May 27 12:05:56 CEST 2016
> [INFO] Final Memory: 119M/797M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>