Re: Review Request 53914: Perf: Fix bugs in deploy-gce-perf-cluster.py to generate correct config file

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53914/#review156380
---


Ship it!




Ship It!

- Sumit Mohanty


On Nov. 19, 2016, 12:12 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53914/
> ---
> 
> (Updated Nov. 19, 2016, 12:12 a.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Dmytro Sen, Sumit Mohanty, Sid 
> Wagle, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18940
> https://issues.apache.org/jira/browse/AMBARI-18940
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The script to create VMs on GCE for the perf cluster has a bug where each VM 
> will have the exact same agent-multiplier config file with start=1 and 
> num=50, thereby preventing more than 50 agents.
> This is because a single server.sh and agent.sh file is written out, and 
> copied to all VMs.
> We need to create a custom config file for each VM instead.
> 
> 
> Diffs
> -
> 
>   contrib/utils/perf/deploy-gce-perf-cluster.py da45f8f 
> 
> Diff: https://reviews.apache.org/r/53914/diff/
> 
> 
> Testing
> ---
> 
> Verified by deploying a cluster to GCE.
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Review Request 53922: AMBARI-18942 - Auto-start services: Avoid full page reload after "Save" / "Discard"

[Richard Zang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53922/
---

Review request for Ambari, Jaimin Jetly, Xi Wang, and Yusaku Sako.


Bugs: AMBARI-18942
https://issues.apache.org/jira/browse/AMBARI-18942


Repository: ambari


Description
---

Avoid reload after save / discard


Diffs
-

  ambari-web/app/controllers/main/admin/service_auto_start.js b425c78 
  ambari-web/app/templates/main/admin/service_auto_start.hbs 9dc4dfb 
  ambari-web/app/views/main/admin/service_auto_start.js dcbbfc9 
  ambari-web/app/views/main/admin/service_auto_start/component_auto_start.js 
a25d835 

Diff: https://reviews.apache.org/r/53922/diff/


Testing
---

Manually tested on live cluster.
All unit tests passed.
  25461 tests complete (22 seconds)
  57 tests pending


Thanks,

Richard Zang

Re: Review Request 53914: Perf: Fix bugs in deploy-gce-perf-cluster.py to generate correct config file

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53914/#review156373
---


Ship it!




Ship It!

- Sid Wagle


On Nov. 19, 2016, 12:12 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53914/
> ---
> 
> (Updated Nov. 19, 2016, 12:12 a.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Dmytro Sen, Sumit Mohanty, Sid 
> Wagle, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18940
> https://issues.apache.org/jira/browse/AMBARI-18940
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The script to create VMs on GCE for the perf cluster has a bug where each VM 
> will have the exact same agent-multiplier config file with start=1 and 
> num=50, thereby preventing more than 50 agents.
> This is because a single server.sh and agent.sh file is written out, and 
> copied to all VMs.
> We need to create a custom config file for each VM instead.
> 
> 
> Diffs
> -
> 
>   contrib/utils/perf/deploy-gce-perf-cluster.py da45f8f 
> 
> Diff: https://reviews.apache.org/r/53914/diff/
> 
> 
> Testing
> ---
> 
> Verified by deploying a cluster to GCE.
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 53801: AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config calculations.

[Swapan Shridhar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53801/
---

(Updated Nov. 19, 2016, 12:49 a.m.)


Review request for Ambari, Alejandro Fernandez, Madhuvanthi Radhakrishnan, 
Sumit Mohanty, and Siddharth Seth.


Changes
---

Removed the unintended change at : 
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java:365.


Bugs: AMBARI-18901
https://issues.apache.org/jira/browse/AMBARI-18901


Repository: ambari


Description
---

AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config 
calculations.

Below is the calculation logic used:


**
**


---
For use with default setup - Ambari managed queue

Parameteres
numRequestedLlapNodes   UserInput
tezAmContainerSize  Computed
memoryPerThread Computed// Set as parameter in 
HiveConf. user can override in Advanced
numConcurrentQueriesComputed// user can override in Advanced
maxConcurrentQueriesComputed// Max value for Slider
numLlapNodesComputed// Can be lower for 
small clusters. user can override in Advanced
sliderAmContainerSize   Computed// user can override in Advanced
numExecutorsPerNode Computed// user can override in Advanced  | 
TODO: This and memPerThread are about the same when taking daemonSize, and 
cache into considertaion
cacheMemoryPerNode  Computed// user can override in 
Advanced
amFraction  1   // Set 
to 1 for Ambari controlled queued. | TODO Factor into concurrency for user 
provided queues
llapQueueFraction   Computed// Computed by Ambari. 
(Avoid changing if the current value is > computed value, and the user 
specified the current value?)


numClusterNodes ClusterParameter
nmMemoryPerNode ClusterParameter
nmCpusPerNode   ClusterParameter
minContainerSizeClusterParameter


CONSTANT DEFAULT_EXECUTOR_TO_AM_RATIO = 20;
CONSTANT MIN_EXECUTOR_TO_AM_RATIO = 10;
CONSTANT MAX_CONCURRENT_QUERIES = 32;

nmMemoryPerNodeNormalized = normalizeDown(nmMemoryPerNode, minContainerSize);

totalClusterCapacity = numClusterNodes * nmMemoryPerNodeNormalized;
totalllapMemory = numRequestedLlapNodes * nmMemoryPerNodeNormalized;
amCapacityAvailable = totalLlapMemory; // For the LLAP queue - the AM fraction 
is set to 1

sliderAmSize -> Current calculations remain unchanged. (<=, < fixes)

llapMemoryTezAMsAndDaemons = totalllapMemory - sliderAmSize;
FAIL("Not enough capacity available on the cluster to run LLAP") if 
(llapMemoryTezAMsAndDaemons < 2 * minContainerSize);

tezAmContainerSize = (totalClusterCapacity, minContainerSize) {
desiredSize = { // This part is unchanged from current calculations.
if (totalClusterCapacity <= 4096) {
return 256;
} else if (totalClusterCapacity <= 73728) {
return 512;
} else {
return 1536;
}
}
return normalizeUp(desiredSize, minContainerSize);
}

memoryPerThread = (nmMemoryPerNodeNormalized, nmCpusPerNode) { // TODO: Not 
linear. e.g. 1024 to 1025 goes from 2 executors to 1.
if (userSpecifiedValue) {
return userSpecifiedValue;
} else if (nmMemoryPerNodeNormalized <= 1024) {
return Math.min(512, nmMemoryPerNodeNormalized)
} else if (nmMemoryPerNodeNormalized <= 4096 ) {
return 1024;
} else if (nmMemoryPerNodeNormalized <= 10240) {
return 2048;
} else if (nmMemoryPerNodeNormalized <= 24576) {
return 3072;
} else {
return 4096;
}
}

numConcurrentQueries, maxConcurrentQueries = (nmMemoryPerNodeNormalized, 
nmCpusPerNode, memoryPerThread, numRequestedLlapNodes, 
llapMemoryTezAMsAndDaemons, tezAmContainerSize, amCapacityAvailable) {
maxExecutorsPerNode = getMaxExecutorsPerNode(nmMemoryPerNodeNormalized, 
nmCpusPerNode, memoryPerThread);
FAIL if maxExecutorsPerNode < 1;


// Default 1 AM for every 20 executor threads.
// The second part of the min calculates based on mem required for 
DEFAULT_EXECUTOR_TO_AM_RATIO executors + 1 AM, making use of total memory. 
However, it's possible
// that total memory will not 

Re: Review Request 53801: AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config calculations.

[Swapan Shridhar]

> On Nov. 18, 2016, 11:57 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java,
> >  line 294
> > 
> >
> > If the property doesn't exist still need to add it.
> > 
> > Same for the 3 other properties below.

Done.


- Swapan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53801/#review156356
---


On Nov. 19, 2016, 12:45 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53801/
> ---
> 
> (Updated Nov. 19, 2016, 12:45 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Madhuvanthi Radhakrishnan, 
> Sumit Mohanty, and Siddharth Seth.
> 
> 
> Bugs: AMBARI-18901
> https://issues.apache.org/jira/browse/AMBARI-18901
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP 
> config calculations.
> 
> Below is the calculation logic used:
> 
> 
> **
> **
> 
> 
> ---
> For use with default setup - Ambari managed queue
> 
> Parameteres
> numRequestedLlapNodes UserInput
> tezAmContainerSizeComputed
> memoryPerThread   Computed// Set as parameter in 
> HiveConf. user can override in Advanced
> numConcurrentQueries  Computed// user can override in Advanced
> maxConcurrentQueries  Computed// Max value for Slider
> numLlapNodes  Computed// Can be lower for 
> small clusters. user can override in Advanced
> sliderAmContainerSize Computed// user can override in Advanced
> numExecutorsPerNode   Computed// user can override in Advanced  | 
> TODO: This and memPerThread are about the same when taking daemonSize, and 
> cache into considertaion
> cacheMemoryPerNodeComputed// user can override in 
> Advanced
> amFraction1   // Set 
> to 1 for Ambari controlled queued. | TODO Factor into concurrency for user 
> provided queues
> llapQueueFraction Computed// Computed by Ambari. 
> (Avoid changing if the current value is > computed value, and the user 
> specified the current value?)
> 
> 
> numClusterNodes   ClusterParameter
> nmMemoryPerNode   ClusterParameter
> nmCpusPerNode ClusterParameter
> minContainerSize  ClusterParameter
> 
> 
> CONSTANT DEFAULT_EXECUTOR_TO_AM_RATIO = 20;
> CONSTANT MIN_EXECUTOR_TO_AM_RATIO = 10;
> CONSTANT MAX_CONCURRENT_QUERIES = 32;
> 
> nmMemoryPerNodeNormalized = normalizeDown(nmMemoryPerNode, minContainerSize);
> 
> totalClusterCapacity = numClusterNodes * nmMemoryPerNodeNormalized;
> totalllapMemory = numRequestedLlapNodes * nmMemoryPerNodeNormalized;
> amCapacityAvailable = totalLlapMemory; // For the LLAP queue - the AM 
> fraction is set to 1
> 
> sliderAmSize -> Current calculations remain unchanged. (<=, < fixes)
> 
> llapMemoryTezAMsAndDaemons = totalllapMemory - sliderAmSize;
> FAIL("Not enough capacity available on the cluster to run LLAP") if 
> (llapMemoryTezAMsAndDaemons < 2 * minContainerSize);
> 
> tezAmContainerSize = (totalClusterCapacity, minContainerSize) {
>   desiredSize = { // This part is unchanged from current calculations.
>   if (totalClusterCapacity <= 4096) {
>   return 256;
>   } else if (totalClusterCapacity <= 73728) {
>   return 512;
>   } else {
>   return 1536;
>   }
>   }
>   return normalizeUp(desiredSize, minContainerSize);
> }
> 
> memoryPerThread = (nmMemoryPerNodeNormalized, nmCpusPerNode) { // TODO: Not 
> linear. e.g. 1024 to 1025 goes from 2 executors to 1.
>   if (userSpecifiedValue) {
>   return userSpecifiedValue;
>   } else if (nmMemoryPerNodeNormalized <= 1024) {
>   return Math.min(512, nmMemoryPerNodeNormalized)
>   } else if (nmMemoryPerNodeNormalized <= 4096 ) {
>   return 1024;
>   } else if (nmMemoryPerNodeNormalized <= 10240) {
>   return 2048;
>   } else if (nmMemoryPerNodeNormalized <= 24576) {
>   return 3072;
>   } 

Re: Review Request 53801: AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config calculations.

[Swapan Shridhar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53801/
---

(Updated Nov. 19, 2016, 12:45 a.m.)


Review request for Ambari, Alejandro Fernandez, Madhuvanthi Radhakrishnan, 
Sumit Mohanty, and Siddharth Seth.


Changes
---

Removed the null check. Configs, whether existing or not, will be added and 
updated with default value.


Bugs: AMBARI-18901
https://issues.apache.org/jira/browse/AMBARI-18901


Repository: ambari


Description
---

AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config 
calculations.

Below is the calculation logic used:


**
**


---
For use with default setup - Ambari managed queue

Parameteres
numRequestedLlapNodes   UserInput
tezAmContainerSize  Computed
memoryPerThread Computed// Set as parameter in 
HiveConf. user can override in Advanced
numConcurrentQueriesComputed// user can override in Advanced
maxConcurrentQueriesComputed// Max value for Slider
numLlapNodesComputed// Can be lower for 
small clusters. user can override in Advanced
sliderAmContainerSize   Computed// user can override in Advanced
numExecutorsPerNode Computed// user can override in Advanced  | 
TODO: This and memPerThread are about the same when taking daemonSize, and 
cache into considertaion
cacheMemoryPerNode  Computed// user can override in 
Advanced
amFraction  1   // Set 
to 1 for Ambari controlled queued. | TODO Factor into concurrency for user 
provided queues
llapQueueFraction   Computed// Computed by Ambari. 
(Avoid changing if the current value is > computed value, and the user 
specified the current value?)


numClusterNodes ClusterParameter
nmMemoryPerNode ClusterParameter
nmCpusPerNode   ClusterParameter
minContainerSizeClusterParameter


CONSTANT DEFAULT_EXECUTOR_TO_AM_RATIO = 20;
CONSTANT MIN_EXECUTOR_TO_AM_RATIO = 10;
CONSTANT MAX_CONCURRENT_QUERIES = 32;

nmMemoryPerNodeNormalized = normalizeDown(nmMemoryPerNode, minContainerSize);

totalClusterCapacity = numClusterNodes * nmMemoryPerNodeNormalized;
totalllapMemory = numRequestedLlapNodes * nmMemoryPerNodeNormalized;
amCapacityAvailable = totalLlapMemory; // For the LLAP queue - the AM fraction 
is set to 1

sliderAmSize -> Current calculations remain unchanged. (<=, < fixes)

llapMemoryTezAMsAndDaemons = totalllapMemory - sliderAmSize;
FAIL("Not enough capacity available on the cluster to run LLAP") if 
(llapMemoryTezAMsAndDaemons < 2 * minContainerSize);

tezAmContainerSize = (totalClusterCapacity, minContainerSize) {
desiredSize = { // This part is unchanged from current calculations.
if (totalClusterCapacity <= 4096) {
return 256;
} else if (totalClusterCapacity <= 73728) {
return 512;
} else {
return 1536;
}
}
return normalizeUp(desiredSize, minContainerSize);
}

memoryPerThread = (nmMemoryPerNodeNormalized, nmCpusPerNode) { // TODO: Not 
linear. e.g. 1024 to 1025 goes from 2 executors to 1.
if (userSpecifiedValue) {
return userSpecifiedValue;
} else if (nmMemoryPerNodeNormalized <= 1024) {
return Math.min(512, nmMemoryPerNodeNormalized)
} else if (nmMemoryPerNodeNormalized <= 4096 ) {
return 1024;
} else if (nmMemoryPerNodeNormalized <= 10240) {
return 2048;
} else if (nmMemoryPerNodeNormalized <= 24576) {
return 3072;
} else {
return 4096;
}
}

numConcurrentQueries, maxConcurrentQueries = (nmMemoryPerNodeNormalized, 
nmCpusPerNode, memoryPerThread, numRequestedLlapNodes, 
llapMemoryTezAMsAndDaemons, tezAmContainerSize, amCapacityAvailable) {
maxExecutorsPerNode = getMaxExecutorsPerNode(nmMemoryPerNodeNormalized, 
nmCpusPerNode, memoryPerThread);
FAIL if maxExecutorsPerNode < 1;


// Default 1 AM for every 20 executor threads.
// The second part of the min calculates based on mem required for 
DEFAULT_EXECUTOR_TO_AM_RATIO executors + 1 AM, making use of total memory. 
However, it's possible
// that total memory will not be used - and the 

Review Request 53914: Perf: Fix bugs in deploy-gce-perf-cluster.py to generate correct config file

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53914/
---

Review request for Ambari, Dmitro Lisnichenko, Dmytro Sen, Sumit Mohanty, Sid 
Wagle, and Vitalyi Brodetskyi.


Bugs: AMBARI-18940
https://issues.apache.org/jira/browse/AMBARI-18940


Repository: ambari


Description
---

The script to create VMs on GCE for the perf cluster has a bug where each VM 
will have the exact same agent-multiplier config file with start=1 and num=50, 
thereby preventing more than 50 agents.
This is because a single server.sh and agent.sh file is written out, and copied 
to all VMs.
We need to create a custom config file for each VM instead.


Diffs
-

  contrib/utils/perf/deploy-gce-perf-cluster.py da45f8f 

Diff: https://reviews.apache.org/r/53914/diff/


Testing
---

Verified by deploying a cluster to GCE.


Thanks,

Alejandro Fernandez

Re: Review Request 53801: AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config calculations.

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53801/#review156356
---


Fix it, then Ship it!





ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
 (line 289)


If the property doesn't exist still need to add it.

Same for the 3 other properties below.


- Alejandro Fernandez


On Nov. 18, 2016, 10:20 p.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53801/
> ---
> 
> (Updated Nov. 18, 2016, 10:20 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Madhuvanthi Radhakrishnan, 
> Sumit Mohanty, and Siddharth Seth.
> 
> 
> Bugs: AMBARI-18901
> https://issues.apache.org/jira/browse/AMBARI-18901
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP 
> config calculations.
> 
> Below is the calculation logic used:
> 
> 
> **
> **
> 
> 
> ---
> For use with default setup - Ambari managed queue
> 
> Parameteres
> numRequestedLlapNodes UserInput
> tezAmContainerSizeComputed
> memoryPerThread   Computed// Set as parameter in 
> HiveConf. user can override in Advanced
> numConcurrentQueries  Computed// user can override in Advanced
> maxConcurrentQueries  Computed// Max value for Slider
> numLlapNodes  Computed// Can be lower for 
> small clusters. user can override in Advanced
> sliderAmContainerSize Computed// user can override in Advanced
> numExecutorsPerNode   Computed// user can override in Advanced  | 
> TODO: This and memPerThread are about the same when taking daemonSize, and 
> cache into considertaion
> cacheMemoryPerNodeComputed// user can override in 
> Advanced
> amFraction1   // Set 
> to 1 for Ambari controlled queued. | TODO Factor into concurrency for user 
> provided queues
> llapQueueFraction Computed// Computed by Ambari. 
> (Avoid changing if the current value is > computed value, and the user 
> specified the current value?)
> 
> 
> numClusterNodes   ClusterParameter
> nmMemoryPerNode   ClusterParameter
> nmCpusPerNode ClusterParameter
> minContainerSize  ClusterParameter
> 
> 
> CONSTANT DEFAULT_EXECUTOR_TO_AM_RATIO = 20;
> CONSTANT MIN_EXECUTOR_TO_AM_RATIO = 10;
> CONSTANT MAX_CONCURRENT_QUERIES = 32;
> 
> nmMemoryPerNodeNormalized = normalizeDown(nmMemoryPerNode, minContainerSize);
> 
> totalClusterCapacity = numClusterNodes * nmMemoryPerNodeNormalized;
> totalllapMemory = numRequestedLlapNodes * nmMemoryPerNodeNormalized;
> amCapacityAvailable = totalLlapMemory; // For the LLAP queue - the AM 
> fraction is set to 1
> 
> sliderAmSize -> Current calculations remain unchanged. (<=, < fixes)
> 
> llapMemoryTezAMsAndDaemons = totalllapMemory - sliderAmSize;
> FAIL("Not enough capacity available on the cluster to run LLAP") if 
> (llapMemoryTezAMsAndDaemons < 2 * minContainerSize);
> 
> tezAmContainerSize = (totalClusterCapacity, minContainerSize) {
>   desiredSize = { // This part is unchanged from current calculations.
>   if (totalClusterCapacity <= 4096) {
>   return 256;
>   } else if (totalClusterCapacity <= 73728) {
>   return 512;
>   } else {
>   return 1536;
>   }
>   }
>   return normalizeUp(desiredSize, minContainerSize);
> }
> 
> memoryPerThread = (nmMemoryPerNodeNormalized, nmCpusPerNode) { // TODO: Not 
> linear. e.g. 1024 to 1025 goes from 2 executors to 1.
>   if (userSpecifiedValue) {
>   return userSpecifiedValue;
>   } else if (nmMemoryPerNodeNormalized <= 1024) {
>   return Math.min(512, nmMemoryPerNodeNormalized)
>   } else if (nmMemoryPerNodeNormalized <= 4096 ) {
>   return 1024;
>   } else if (nmMemoryPerNodeNormalized <= 10240) {
>   return 2048;
>   } else if (nmMemoryPerNodeNormalized <= 24576) {
>   return 3072;
>   } else {
>   return 4096;
>   }
> }
> 
> numConcurrentQueries, 

Re: Review Request 53881: Remove Unnecessary Locks Inside Of ConfigGroup Business Object Implementations

[Jonathan Hurley]

> On Nov. 18, 2016, 6:21 p.m., Sid Wagle wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java,
> >  line 402
> > 
> >
> > private modifier will not allow Transaction annotations from getting 
> > chained.

Really ... I thought that the way in which we bind @Transactional allows us to 
get around this. With that said, I'm curious why @Transactional doesn't warn on 
this (or we don't have a test for it). 

`# grep -r --include "*.java" "@Transactional" src/main -A1 | grep private | wc 
-l`
`14`

So we have 14 instances of using @Transactional with private methods.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53881/#review156352
---


On Nov. 18, 2016, 6:37 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53881/
> ---
> 
> (Updated Nov. 18, 2016, 6:37 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-18933
> https://issues.apache.org/jira/browse/AMBARI-18933
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> {{ConfigGroupImpl}} uses internal locks around state which can lead to 
> deadlocks when this in called in the context of other business objects. In 
> most cases, this state-full data does not need locks placed around it. 
> 
> {{ConfigGroupImpl}} should be changed to:
> - No longer store entity information
> - No longer require locks around primitive state
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigGroupResourceProvider.java
>  b957f0a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ConfigImpl.java 
> e68839f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  8b157c7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroup.java
>  1b29c9b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupFactory.java
>  9abadf3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
>  9a2fc88 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  ed43ee1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  098efa9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigGroupTest.java
>  75853db 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
>  a3a7e11 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterTest.java
>  69cfc9f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
>  8db5190 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  68a8d4c 
> 
> Diff: https://reviews.apache.org/r/53881/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4772, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 29:17 min
> [INFO] Finished at: 2016-11-18T14:02:44-05:00
> [INFO] Final Memory: 57M/731M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 53881: Remove Unnecessary Locks Inside Of ConfigGroup Business Object Implementations

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53881/
---

(Updated Nov. 18, 2016, 6:37 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and 
Sid Wagle.


Changes
---

Adjusted for Sid's comments.


Bugs: AMBARI-18933
https://issues.apache.org/jira/browse/AMBARI-18933


Repository: ambari


Description
---

{{ConfigGroupImpl}} uses internal locks around state which can lead to 
deadlocks when this in called in the context of other business objects. In most 
cases, this state-full data does not need locks placed around it. 

{{ConfigGroupImpl}} should be changed to:
- No longer store entity information
- No longer require locks around primitive state


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigGroupResourceProvider.java
 b957f0a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ConfigImpl.java 
e68839f 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 8b157c7 
  
ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroup.java
 1b29c9b 
  
ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupFactory.java
 9abadf3 
  
ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
 9a2fc88 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
 ed43ee1 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
 098efa9 
  
ambari-server/src/test/java/org/apache/ambari/server/state/ConfigGroupTest.java 
75853db 
  
ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
 a3a7e11 
  
ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterTest.java
 69cfc9f 
  
ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
 8db5190 
  
ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
 68a8d4c 

Diff: https://reviews.apache.org/r/53881/diff/


Testing
---

Tests run: 4772, Failures: 0, Errors: 0, Skipped: 37

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 29:17 min
[INFO] Finished at: 2016-11-18T14:02:44-05:00
[INFO] Final Memory: 57M/731M
[INFO] 


Thanks,

Jonathan Hurley

Re: Review Request 53881: Remove Unnecessary Locks Inside Of ConfigGroup Business Object Implementations

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53881/#review156352
---




ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
 (line 328)


private modifier will not allow Transaction annotations from getting 
chained.



ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
 (line 348)


Similar issue here.


- Sid Wagle


On Nov. 18, 2016, 8:50 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53881/
> ---
> 
> (Updated Nov. 18, 2016, 8:50 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-18933
> https://issues.apache.org/jira/browse/AMBARI-18933
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> {{ConfigGroupImpl}} uses internal locks around state which can lead to 
> deadlocks when this in called in the context of other business objects. In 
> most cases, this state-full data does not need locks placed around it. 
> 
> {{ConfigGroupImpl}} should be changed to:
> - No longer store entity information
> - No longer require locks around primitive state
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigGroupResourceProvider.java
>  b957f0a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  8b157c7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroup.java
>  1b29c9b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupFactory.java
>  9abadf3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
>  9a2fc88 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  ed43ee1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  098efa9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigGroupTest.java
>  75853db 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
>  a3a7e11 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterTest.java
>  69cfc9f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
>  8db5190 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  68a8d4c 
> 
> Diff: https://reviews.apache.org/r/53881/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4772, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 29:17 min
> [INFO] Finished at: 2016-11-18T14:02:44-05:00
> [INFO] Final Memory: 57M/731M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 53907: Add Service wizard: Allow adding slave components from different service conditionally

[Andriy Babiichuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53907/#review156350
---


Ship it!




Ship It!

- Andriy Babiichuk


On Лис. 18, 2016, 11:04 після полудня, Aleksandr Kovalenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53907/
> ---
> 
> (Updated Лис. 18, 2016, 11:04 після полудня)
> 
> 
> Review request for Ambari, Alexandr Antonenko, Jaimin Jetly, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-18939
> https://issues.apache.org/jira/browse/AMBARI-18939
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> Add LIVY_SERVER as serviceComponent dependency to "ZEPPELIN_MASTER" in 
> Zeppelin's service definiion
> Add Spark without Livy (slave component with cardinality of 0+)
> Use "Add Service Wizard" to add Zeppelin service.
> Expected Result with new work to be done: User should be able to edit 
> (add/change) slave dependencies (LIVY_SERVER) as dependency rule is broken 
> while adding Zeppelin service i.e. Zeppelin mater depends on livy server and 
> there is no livy server installed on deployed cluster.
> Actual Result as of now: "Assign Slave and Client component" page is skipped 
> while adding zeppelin
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/main/service/add_controller.js 04427a2 
>   ambari-web/app/controllers/wizard.js 89d439a 
>   ambari-web/app/controllers/wizard/step6_controller.js 87b0ee5 
>   ambari-web/app/mappers/stack_service_mapper.js 888dcdf 
>   ambari-web/app/utils/ajax/ajax.js 6ca014a 
>   ambari-web/test/controllers/main/service/add_controller_test.js 51c8dbf 
>   ambari-web/test/controllers/wizard/step6_test.js 4cf155c 
> 
> Diff: https://reviews.apache.org/r/53907/diff/
> 
> 
> Testing
> ---
> 
> 30624 tests complete (33 seconds)
>   153 tests pending
> 
> 
> Thanks,
> 
> Aleksandr Kovalenko
> 
>

Review Request 53907: Add Service wizard: Allow adding slave components from different service conditionally

[Aleksandr Kovalenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53907/
---

Review request for Ambari, Alexandr Antonenko, Jaimin Jetly, and Yusaku Sako.


Bugs: AMBARI-18939
https://issues.apache.org/jira/browse/AMBARI-18939


Repository: ambari


Description
---

STR:
Add LIVY_SERVER as serviceComponent dependency to "ZEPPELIN_MASTER" in 
Zeppelin's service definiion
Add Spark without Livy (slave component with cardinality of 0+)
Use "Add Service Wizard" to add Zeppelin service.
Expected Result with new work to be done: User should be able to edit 
(add/change) slave dependencies (LIVY_SERVER) as dependency rule is broken 
while adding Zeppelin service i.e. Zeppelin mater depends on livy server and 
there is no livy server installed on deployed cluster.
Actual Result as of now: "Assign Slave and Client component" page is skipped 
while adding zeppelin


Diffs
-

  ambari-web/app/controllers/main/service/add_controller.js 04427a2 
  ambari-web/app/controllers/wizard.js 89d439a 
  ambari-web/app/controllers/wizard/step6_controller.js 87b0ee5 
  ambari-web/app/mappers/stack_service_mapper.js 888dcdf 
  ambari-web/app/utils/ajax/ajax.js 6ca014a 
  ambari-web/test/controllers/main/service/add_controller_test.js 51c8dbf 
  ambari-web/test/controllers/wizard/step6_test.js 4cf155c 

Diff: https://reviews.apache.org/r/53907/diff/


Testing
---

30624 tests complete (33 seconds)
  153 tests pending


Thanks,

Aleksandr Kovalenko

Review Request 53904: Implement Create Alerts: step 1 select alert type

[Xi Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53904/
---

Review request for Ambari, Richard Zang, Vivek Ratnavel Subramanian, and Yusaku 
Sako.


Bugs: AMBARI-18912
https://issues.apache.org/jira/browse/AMBARI-18912


Repository: ambari


Description
---

This is a FE task to implement the "Create Alerts Wizard " based on the design 
attached.
In this task, should implement the step 1 in the design. All available alert 
types should be displayed as options with icon, type and description. Make sure 
that on clicking a type use would jump to next step.


Diffs
-

  
ambari-web/app/controllers/main/alerts/add_alert_definition/step1_controller.js 
2d4f338 
  ambari-web/app/messages.js 7ba19db 
  ambari-web/app/models/alerts/alert_definition.js a3c8850 
  ambari-web/app/styles/alerts.less 4569e25 
  ambari-web/app/templates/main/alerts/add_alert_definition/step1.hbs ee61648 
  ambari-web/app/views/main/alerts/add_alert_definition/step1_view.js d55d20f 

Diff: https://reviews.apache.org/r/53904/diff/


Testing
---


Thanks,

Xi Wang

Re: Review Request 53901: Allow generic Service Upgrade Packs for all targets

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53901/#review156346
---


Ship it!




Ship It!

- Jonathan Hurley


On Nov. 18, 2016, 4:12 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53901/
> ---
> 
> (Updated Nov. 18, 2016, 4:12 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Jayush 
> Luniya.
> 
> 
> Bugs: AMBARI-18937
> https://issues.apache.org/jira/browse/AMBARI-18937
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, a custom service can participate in RU/EU only if it adds an 
> Upgrade Pack for every target of a stack.  If there is a case of the Upgrade 
> Pack ALWAYS being the same no matter the version, then only one file should 
> be sufficient.
> 
> Also added that a group name must be unique in a XSD since the logic already 
> dictated that it must be true (and associated fixes).
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> 37f4167 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
>  edf5c89 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 95c5f06 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml 
> a01996a 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> f0c6131 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 
> f520faf 
>   ambari-server/src/main/resources/upgrade-pack.xsd 2871f05 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/stack/UpgradePackTest.java
>  dde7ffa 
>   
> ambari-server/src/test/resources/stacks/HDP/2.2.0/services/GANGLIA/upgrades/HDP/rolling-upgrade.xml
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53901/diff/
> 
> 
> Testing
> ---
> 
> Manual.  Automated pending.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 53801: AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config calculations.

[Swapan Shridhar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53801/
---

(Updated Nov. 18, 2016, 10:20 p.m.)


Review request for Ambari, Alejandro Fernandez, Madhuvanthi Radhakrishnan, 
Sumit Mohanty, and Siddharth Seth.


Changes
---

Version 3 changes are scrapped, as they were intended. Infact, same change was 
required for config 'tez.am.resource.memory.mb'.
Removed the following:

Earlier we wanted hive-interactive-site's 'tez.am.resource.memory.mb' config to 
be set only once 1st time (during initialization) by stack Advisor if the 
config value is "SET_ON_FIRST_INVOCATION".
That requirement is no more valid. Thus, removing code and test for it. The 
config would not be set everytime change is detected for the config calculation.


Bugs: AMBARI-18901
https://issues.apache.org/jira/browse/AMBARI-18901


Repository: ambari


Description
---

AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config 
calculations.

Below is the calculation logic used:


**
**


---
For use with default setup - Ambari managed queue

Parameteres
numRequestedLlapNodes   UserInput
tezAmContainerSize  Computed
memoryPerThread Computed// Set as parameter in 
HiveConf. user can override in Advanced
numConcurrentQueriesComputed// user can override in Advanced
maxConcurrentQueriesComputed// Max value for Slider
numLlapNodesComputed// Can be lower for 
small clusters. user can override in Advanced
sliderAmContainerSize   Computed// user can override in Advanced
numExecutorsPerNode Computed// user can override in Advanced  | 
TODO: This and memPerThread are about the same when taking daemonSize, and 
cache into considertaion
cacheMemoryPerNode  Computed// user can override in 
Advanced
amFraction  1   // Set 
to 1 for Ambari controlled queued. | TODO Factor into concurrency for user 
provided queues
llapQueueFraction   Computed// Computed by Ambari. 
(Avoid changing if the current value is > computed value, and the user 
specified the current value?)


numClusterNodes ClusterParameter
nmMemoryPerNode ClusterParameter
nmCpusPerNode   ClusterParameter
minContainerSizeClusterParameter


CONSTANT DEFAULT_EXECUTOR_TO_AM_RATIO = 20;
CONSTANT MIN_EXECUTOR_TO_AM_RATIO = 10;
CONSTANT MAX_CONCURRENT_QUERIES = 32;

nmMemoryPerNodeNormalized = normalizeDown(nmMemoryPerNode, minContainerSize);

totalClusterCapacity = numClusterNodes * nmMemoryPerNodeNormalized;
totalllapMemory = numRequestedLlapNodes * nmMemoryPerNodeNormalized;
amCapacityAvailable = totalLlapMemory; // For the LLAP queue - the AM fraction 
is set to 1

sliderAmSize -> Current calculations remain unchanged. (<=, < fixes)

llapMemoryTezAMsAndDaemons = totalllapMemory - sliderAmSize;
FAIL("Not enough capacity available on the cluster to run LLAP") if 
(llapMemoryTezAMsAndDaemons < 2 * minContainerSize);

tezAmContainerSize = (totalClusterCapacity, minContainerSize) {
desiredSize = { // This part is unchanged from current calculations.
if (totalClusterCapacity <= 4096) {
return 256;
} else if (totalClusterCapacity <= 73728) {
return 512;
} else {
return 1536;
}
}
return normalizeUp(desiredSize, minContainerSize);
}

memoryPerThread = (nmMemoryPerNodeNormalized, nmCpusPerNode) { // TODO: Not 
linear. e.g. 1024 to 1025 goes from 2 executors to 1.
if (userSpecifiedValue) {
return userSpecifiedValue;
} else if (nmMemoryPerNodeNormalized <= 1024) {
return Math.min(512, nmMemoryPerNodeNormalized)
} else if (nmMemoryPerNodeNormalized <= 4096 ) {
return 1024;
} else if (nmMemoryPerNodeNormalized <= 10240) {
return 2048;
} else if (nmMemoryPerNodeNormalized <= 24576) {
return 3072;
} else {
return 4096;
}
}

numConcurrentQueries, maxConcurrentQueries = (nmMemoryPerNodeNormalized, 
nmCpusPerNode, memoryPerThread, numRequestedLlapNodes, 
llapMemoryTezAMsAndDaemons, tezAmContainerSize, amCapacityAvailable) {
maxExecutorsPerNode = 

Re: Review Request 53881: Remove Unnecessary Locks Inside Of ConfigGroup Business Object Implementations

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53881/#review156345
---


Ship it!




Ship It!

- Alejandro Fernandez


On Nov. 18, 2016, 8:50 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53881/
> ---
> 
> (Updated Nov. 18, 2016, 8:50 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-18933
> https://issues.apache.org/jira/browse/AMBARI-18933
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> {{ConfigGroupImpl}} uses internal locks around state which can lead to 
> deadlocks when this in called in the context of other business objects. In 
> most cases, this state-full data does not need locks placed around it. 
> 
> {{ConfigGroupImpl}} should be changed to:
> - No longer store entity information
> - No longer require locks around primitive state
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigGroupResourceProvider.java
>  b957f0a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  8b157c7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroup.java
>  1b29c9b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupFactory.java
>  9abadf3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
>  9a2fc88 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  ed43ee1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  098efa9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigGroupTest.java
>  75853db 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
>  a3a7e11 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterTest.java
>  69cfc9f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
>  8db5190 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  68a8d4c 
> 
> Diff: https://reviews.apache.org/r/53881/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4772, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 29:17 min
> [INFO] Finished at: 2016-11-18T14:02:44-05:00
> [INFO] Final Memory: 57M/731M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 53901: Allow generic Service Upgrade Packs for all targets

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53901/#review156344
---


Ship it!




Ship It!

- Alejandro Fernandez


On Nov. 18, 2016, 9:12 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53901/
> ---
> 
> (Updated Nov. 18, 2016, 9:12 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Jayush 
> Luniya.
> 
> 
> Bugs: AMBARI-18937
> https://issues.apache.org/jira/browse/AMBARI-18937
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, a custom service can participate in RU/EU only if it adds an 
> Upgrade Pack for every target of a stack.  If there is a case of the Upgrade 
> Pack ALWAYS being the same no matter the version, then only one file should 
> be sufficient.
> 
> Also added that a group name must be unique in a XSD since the logic already 
> dictated that it must be true (and associated fixes).
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> 37f4167 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
>  edf5c89 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 95c5f06 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml 
> a01996a 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> f0c6131 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 
> f520faf 
>   ambari-server/src/main/resources/upgrade-pack.xsd 2871f05 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/stack/UpgradePackTest.java
>  dde7ffa 
>   
> ambari-server/src/test/resources/stacks/HDP/2.2.0/services/GANGLIA/upgrades/HDP/rolling-upgrade.xml
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53901/diff/
> 
> 
> Testing
> ---
> 
> Manual.  Automated pending.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 53901: Allow generic Service Upgrade Packs for all targets

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53901/#review156342
---




ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
(lines 909 - 910)


Is this really a warning? It's not a problem if CUSTOMSERVICE says it wants 
to be after Kafka and Kafka isn't installed.

Or is this a larger problem if "How does a service ensure it's always 
upgraded"? The service it chooses might not exist.


- Jonathan Hurley


On Nov. 18, 2016, 4:12 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53901/
> ---
> 
> (Updated Nov. 18, 2016, 4:12 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Jayush 
> Luniya.
> 
> 
> Bugs: AMBARI-18937
> https://issues.apache.org/jira/browse/AMBARI-18937
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently, a custom service can participate in RU/EU only if it adds an 
> Upgrade Pack for every target of a stack.  If there is a case of the Upgrade 
> Pack ALWAYS being the same no matter the version, then only one file should 
> be sufficient.
> 
> Also added that a group name must be unique in a XSD since the logic already 
> dictated that it must be true (and associated fixes).
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> 37f4167 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
>  edf5c89 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
> 95c5f06 
>   ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml 
> a01996a 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
> f0c6131 
>   ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 
> f520faf 
>   ambari-server/src/main/resources/upgrade-pack.xsd 2871f05 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/stack/UpgradePackTest.java
>  dde7ffa 
>   
> ambari-server/src/test/resources/stacks/HDP/2.2.0/services/GANGLIA/upgrades/HDP/rolling-upgrade.xml
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/53901/diff/
> 
> 
> Testing
> ---
> 
> Manual.  Automated pending.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 53881: Remove Unnecessary Locks Inside Of ConfigGroup Business Object Implementations

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53881/#review156340
---


Ship it!




Ship It!

- Nate Cole


On Nov. 18, 2016, 3:50 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53881/
> ---
> 
> (Updated Nov. 18, 2016, 3:50 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-18933
> https://issues.apache.org/jira/browse/AMBARI-18933
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> {{ConfigGroupImpl}} uses internal locks around state which can lead to 
> deadlocks when this in called in the context of other business objects. In 
> most cases, this state-full data does not need locks placed around it. 
> 
> {{ConfigGroupImpl}} should be changed to:
> - No longer store entity information
> - No longer require locks around primitive state
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigGroupResourceProvider.java
>  b957f0a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  8b157c7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroup.java
>  1b29c9b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupFactory.java
>  9abadf3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
>  9a2fc88 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  ed43ee1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  098efa9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigGroupTest.java
>  75853db 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
>  a3a7e11 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterTest.java
>  69cfc9f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
>  8db5190 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  68a8d4c 
> 
> Diff: https://reviews.apache.org/r/53881/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4772, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 29:17 min
> [INFO] Finished at: 2016-11-18T14:02:44-05:00
> [INFO] Final Memory: 57M/731M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Review Request 53901: Allow generic Service Upgrade Packs for all targets

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53901/
---

Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, and Jayush 
Luniya.


Bugs: AMBARI-18937
https://issues.apache.org/jira/browse/AMBARI-18937


Repository: ambari


Description
---

Currently, a custom service can participate in RU/EU only if it adds an Upgrade 
Pack for every target of a stack.  If there is a case of the Upgrade Pack 
ALWAYS being the same no matter the version, then only one file should be 
sufficient.

Also added that a group name must be unique in a XSD since the logic already 
dictated that it must be true (and associated fixes).


Diffs
-

  ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
37f4167 
  
ambari-server/src/main/java/org/apache/ambari/server/state/stack/UpgradePack.java
 edf5c89 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml 
95c5f06 
  ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml 
a01996a 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml 
f0c6131 
  ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 
f520faf 
  ambari-server/src/main/resources/upgrade-pack.xsd 2871f05 
  
ambari-server/src/test/java/org/apache/ambari/server/state/stack/UpgradePackTest.java
 dde7ffa 
  
ambari-server/src/test/resources/stacks/HDP/2.2.0/services/GANGLIA/upgrades/HDP/rolling-upgrade.xml
 PRE-CREATION 

Diff: https://reviews.apache.org/r/53901/diff/


Testing
---

Manual.  Automated pending.


Thanks,

Nate Cole

Re: Review Request 53871: Perf: Add Hadoop Core services to PERF stack

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53871/#review156339
---


Ship it!




Great, pretty much a self contained change.

- Sumit Mohanty


On Nov. 18, 2016, 1:56 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53871/
> ---
> 
> (Updated Nov. 18, 2016, 1:56 a.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Dmytro Sen, Myroslav 
> Papirkovskyy, Sumit Mohanty, Sid Wagle, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-18928
> https://issues.apache.org/jira/browse/AMBARI-18928
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Right now the PERF stack has dummy services, but we need to add more 
> complexity with services like Hadoop Core (HDFS, YARN, MR, ZK, HBASE) that 
> have more config types and attributes.
> 
> I combined all of the scripts from common services, HDP 2.0.6, 2.1, 2.2, 2.3, 
> 2.4, 2.5, and 2.6 with the correct overrides for HDFS, YARN/MR, ZK, and HBASE 
> to make a single stack version.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/configuration/grumpy-site.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/metainfo.xml 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/package/scripts/dwarf.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/package/scripts/service_check.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/GRUMPY/themes/theme.json
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/alerts.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-env.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-log4j.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-logsearch-conf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-policy.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/hbase-site.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-audit.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-policymgr-ssl.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/configuration/ranger-hbase-security.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/kerberos.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metainfo.xml 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/metrics.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_client.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_master.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/hbase_regionserver.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/phoenix_queryserver.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/package/scripts/service_check.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/quicklinks/quicklinks.json
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/themes/theme.json
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HBASE/widgets.json 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/alerts.json 
> PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/core-site.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-env.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-metrics2.properties.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hadoop-policy.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/HDFS/configuration/hdfs-log4j.xml
>  PRE-CREATION 
>   
> 

Re: Review Request 53881: Remove Unnecessary Locks Inside Of ConfigGroup Business Object Implementations

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53881/#review156338
---




ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroup.java
 


I changed this business object to follow our other objects WRT merging. Now 
you don't need an explicit call to merge after the add.

This was dangerous anyway since it could cause the in-memory state to be 
out of sync.



ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
 (lines 69 - 73)


It's unfortunate, but based on how we store/manipulate mapped hosts, it's 
possible that concurrent calls could cause the internal state to get out of 
sync. So, we need to use this on updating the mapped hosts (but not on read)


- Jonathan Hurley


On Nov. 18, 2016, 3:50 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53881/
> ---
> 
> (Updated Nov. 18, 2016, 3:50 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-18933
> https://issues.apache.org/jira/browse/AMBARI-18933
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> {{ConfigGroupImpl}} uses internal locks around state which can lead to 
> deadlocks when this in called in the context of other business objects. In 
> most cases, this state-full data does not need locks placed around it. 
> 
> {{ConfigGroupImpl}} should be changed to:
> - No longer store entity information
> - No longer require locks around primitive state
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigGroupResourceProvider.java
>  b957f0a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  8b157c7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroup.java
>  1b29c9b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupFactory.java
>  9abadf3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
>  9a2fc88 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  ed43ee1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  098efa9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigGroupTest.java
>  75853db 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
>  a3a7e11 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterTest.java
>  69cfc9f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
>  8db5190 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  68a8d4c 
> 
> Diff: https://reviews.apache.org/r/53881/diff/
> 
> 
> Testing
> ---
> 
> Tests run: 4772, Failures: 0, Errors: 0, Skipped: 37
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 29:17 min
> [INFO] Finished at: 2016-11-18T14:02:44-05:00
> [INFO] Final Memory: 57M/731M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 53881: Remove Unnecessary Locks Inside Of ConfigGroup Business Object Implementations

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53881/
---

(Updated Nov. 18, 2016, 3:50 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, Robert Levas, and 
Sid Wagle.


Changes
---

Updated with working tests.


Bugs: AMBARI-18933
https://issues.apache.org/jira/browse/AMBARI-18933


Repository: ambari


Description
---

{{ConfigGroupImpl}} uses internal locks around state which can lead to 
deadlocks when this in called in the context of other business objects. In most 
cases, this state-full data does not need locks placed around it. 

{{ConfigGroupImpl}} should be changed to:
- No longer store entity information
- No longer require locks around primitive state


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigGroupResourceProvider.java
 b957f0a 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 8b157c7 
  
ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroup.java
 1b29c9b 
  
ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupFactory.java
 9abadf3 
  
ambari-server/src/main/java/org/apache/ambari/server/state/configgroup/ConfigGroupImpl.java
 9a2fc88 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
 ed43ee1 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
 098efa9 
  
ambari-server/src/test/java/org/apache/ambari/server/state/ConfigGroupTest.java 
75853db 
  
ambari-server/src/test/java/org/apache/ambari/server/state/ConfigHelperTest.java
 a3a7e11 
  
ambari-server/src/test/java/org/apache/ambari/server/state/cluster/ClusterTest.java
 69cfc9f 
  
ambari-server/src/test/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostTest.java
 8db5190 
  
ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
 68a8d4c 

Diff: https://reviews.apache.org/r/53881/diff/


Testing (updated)
---

Tests run: 4772, Failures: 0, Errors: 0, Skipped: 37

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 29:17 min
[INFO] Finished at: 2016-11-18T14:02:44-05:00
[INFO] Final Memory: 57M/731M
[INFO] 


Thanks,

Jonathan Hurley

Re: Review Request 53801: AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config calculations.

[Swapan Shridhar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53801/
---

(Updated Nov. 18, 2016, 8:20 p.m.)


Review request for Ambari, Alejandro Fernandez, Madhuvanthi Radhakrishnan, 
Sumit Mohanty, and Siddharth Seth.


Changes
---

Removed the following:

- Earlier we wanted hive-interactive-site's 'hive.tez.container.size' config to 
be set only once 1st time (during initialization) by stack Advisor if the 
config value is "SET_ON_FIRST_INVOCATION". 
- That requirement is no more valid. Thus, removing code and test for it. The 
config would not be set everytime change is detected for the config calculation.


Bugs: AMBARI-18901
https://issues.apache.org/jira/browse/AMBARI-18901


Repository: ambari


Description
---

AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config 
calculations.

Below is the calculation logic used:


**
**


---
For use with default setup - Ambari managed queue

Parameteres
numRequestedLlapNodes   UserInput
tezAmContainerSize  Computed
memoryPerThread Computed// Set as parameter in 
HiveConf. user can override in Advanced
numConcurrentQueriesComputed// user can override in Advanced
maxConcurrentQueriesComputed// Max value for Slider
numLlapNodesComputed// Can be lower for 
small clusters. user can override in Advanced
sliderAmContainerSize   Computed// user can override in Advanced
numExecutorsPerNode Computed// user can override in Advanced  | 
TODO: This and memPerThread are about the same when taking daemonSize, and 
cache into considertaion
cacheMemoryPerNode  Computed// user can override in 
Advanced
amFraction  1   // Set 
to 1 for Ambari controlled queued. | TODO Factor into concurrency for user 
provided queues
llapQueueFraction   Computed// Computed by Ambari. 
(Avoid changing if the current value is > computed value, and the user 
specified the current value?)


numClusterNodes ClusterParameter
nmMemoryPerNode ClusterParameter
nmCpusPerNode   ClusterParameter
minContainerSizeClusterParameter


CONSTANT DEFAULT_EXECUTOR_TO_AM_RATIO = 20;
CONSTANT MIN_EXECUTOR_TO_AM_RATIO = 10;
CONSTANT MAX_CONCURRENT_QUERIES = 32;

nmMemoryPerNodeNormalized = normalizeDown(nmMemoryPerNode, minContainerSize);

totalClusterCapacity = numClusterNodes * nmMemoryPerNodeNormalized;
totalllapMemory = numRequestedLlapNodes * nmMemoryPerNodeNormalized;
amCapacityAvailable = totalLlapMemory; // For the LLAP queue - the AM fraction 
is set to 1

sliderAmSize -> Current calculations remain unchanged. (<=, < fixes)

llapMemoryTezAMsAndDaemons = totalllapMemory - sliderAmSize;
FAIL("Not enough capacity available on the cluster to run LLAP") if 
(llapMemoryTezAMsAndDaemons < 2 * minContainerSize);

tezAmContainerSize = (totalClusterCapacity, minContainerSize) {
desiredSize = { // This part is unchanged from current calculations.
if (totalClusterCapacity <= 4096) {
return 256;
} else if (totalClusterCapacity <= 73728) {
return 512;
} else {
return 1536;
}
}
return normalizeUp(desiredSize, minContainerSize);
}

memoryPerThread = (nmMemoryPerNodeNormalized, nmCpusPerNode) { // TODO: Not 
linear. e.g. 1024 to 1025 goes from 2 executors to 1.
if (userSpecifiedValue) {
return userSpecifiedValue;
} else if (nmMemoryPerNodeNormalized <= 1024) {
return Math.min(512, nmMemoryPerNodeNormalized)
} else if (nmMemoryPerNodeNormalized <= 4096 ) {
return 1024;
} else if (nmMemoryPerNodeNormalized <= 10240) {
return 2048;
} else if (nmMemoryPerNodeNormalized <= 24576) {
return 3072;
} else {
return 4096;
}
}

numConcurrentQueries, maxConcurrentQueries = (nmMemoryPerNodeNormalized, 
nmCpusPerNode, memoryPerThread, numRequestedLlapNodes, 
llapMemoryTezAMsAndDaemons, tezAmContainerSize, amCapacityAvailable) {
maxExecutorsPerNode = getMaxExecutorsPerNode(nmMemoryPerNodeNormalized, 
nmCpusPerNode, memoryPerThread);
FAIL if maxExecutorsPerNode < 1;


// 

Re: Review Request 53880: Can't change capacity-scheduler's queue capacity from the YARN config page, even though its shown as text box

[Alexandr Antonenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53880/#review156334
---


Ship it!




Ship It!

- Alexandr Antonenko


On Nov. 18, 2016, 4:52 p.m., Aleksandr Kovalenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53880/
> ---
> 
> (Updated Nov. 18, 2016, 4:52 p.m.)
> 
> 
> Review request for Ambari, Andrii Tkach, Jaimin Jetly, Oleg Nechiporenko, and 
> Yusaku Sako.
> 
> 
> Bugs: AMBARI-18931
> https://issues.apache.org/jira/browse/AMBARI-18931
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If one tries to change the capacity-scheduler from YARN-> config-> Scheduler, 
> the recommendation call kicks in and overwrites what user intended to do.
> 
> 
> Diffs
> -
> 
>   ambari-web/app/mixins/common/configs/config_recommendation_parser.js 
> b88cf62 
>   ambari-web/app/mixins/common/configs/config_recommendations.js 79d6f5d 
>   ambari-web/app/mixins/common/configs/enhanced_configs.js f53b45d 
>   ambari-web/app/views/common/controls_view.js 955fa80 
> 
> Diff: https://reviews.apache.org/r/53880/diff/
> 
> 
> Testing
> ---
> 
> 30621 tests complete (34 seconds)
>   151 tests pending
> 
> 
> Thanks,
> 
> Aleksandr Kovalenko
> 
>

Re: Review Request 53801: AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP config calculations.

[Siddharth Seth]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53801/#review156328
---


Ship it!




+1 for the stack_advisor computations.

- Siddharth Seth


On Nov. 18, 2016, 2:34 a.m., Swapan Shridhar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53801/
> ---
> 
> (Updated Nov. 18, 2016, 2:34 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Madhuvanthi Radhakrishnan, 
> Sumit Mohanty, and Siddharth Seth.
> 
> 
> Bugs: AMBARI-18901
> https://issues.apache.org/jira/browse/AMBARI-18901
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-18901. Use 'Number of LLAP Nodes' selected as the driver for LLAP 
> config calculations.
> 
> Below is the calculation logic used:
> 
> 
> **
> **
> 
> 
> ---
> For use with default setup - Ambari managed queue
> 
> Parameteres
> numRequestedLlapNodes UserInput
> tezAmContainerSizeComputed
> memoryPerThread   Computed// Set as parameter in 
> HiveConf. user can override in Advanced
> numConcurrentQueries  Computed// user can override in Advanced
> maxConcurrentQueries  Computed// Max value for Slider
> numLlapNodes  Computed// Can be lower for 
> small clusters. user can override in Advanced
> sliderAmContainerSize Computed// user can override in Advanced
> numExecutorsPerNode   Computed// user can override in Advanced  | 
> TODO: This and memPerThread are about the same when taking daemonSize, and 
> cache into considertaion
> cacheMemoryPerNodeComputed// user can override in 
> Advanced
> amFraction1   // Set 
> to 1 for Ambari controlled queued. | TODO Factor into concurrency for user 
> provided queues
> llapQueueFraction Computed// Computed by Ambari. 
> (Avoid changing if the current value is > computed value, and the user 
> specified the current value?)
> 
> 
> numClusterNodes   ClusterParameter
> nmMemoryPerNode   ClusterParameter
> nmCpusPerNode ClusterParameter
> minContainerSize  ClusterParameter
> 
> 
> CONSTANT DEFAULT_EXECUTOR_TO_AM_RATIO = 20;
> CONSTANT MIN_EXECUTOR_TO_AM_RATIO = 10;
> CONSTANT MAX_CONCURRENT_QUERIES = 32;
> 
> nmMemoryPerNodeNormalized = normalizeDown(nmMemoryPerNode, minContainerSize);
> 
> totalClusterCapacity = numClusterNodes * nmMemoryPerNodeNormalized;
> totalllapMemory = numRequestedLlapNodes * nmMemoryPerNodeNormalized;
> amCapacityAvailable = totalLlapMemory; // For the LLAP queue - the AM 
> fraction is set to 1
> 
> sliderAmSize -> Current calculations remain unchanged. (<=, < fixes)
> 
> llapMemoryTezAMsAndDaemons = totalllapMemory - sliderAmSize;
> FAIL("Not enough capacity available on the cluster to run LLAP") if 
> (llapMemoryTezAMsAndDaemons < 2 * minContainerSize);
> 
> tezAmContainerSize = (totalClusterCapacity, minContainerSize) {
>   desiredSize = { // This part is unchanged from current calculations.
>   if (totalClusterCapacity <= 4096) {
>   return 256;
>   } else if (totalClusterCapacity <= 73728) {
>   return 512;
>   } else {
>   return 1536;
>   }
>   }
>   return normalizeUp(desiredSize, minContainerSize);
> }
> 
> memoryPerThread = (nmMemoryPerNodeNormalized, nmCpusPerNode) { // TODO: Not 
> linear. e.g. 1024 to 1025 goes from 2 executors to 1.
>   if (userSpecifiedValue) {
>   return userSpecifiedValue;
>   } else if (nmMemoryPerNodeNormalized <= 1024) {
>   return Math.min(512, nmMemoryPerNodeNormalized)
>   } else if (nmMemoryPerNodeNormalized <= 4096 ) {
>   return 1024;
>   } else if (nmMemoryPerNodeNormalized <= 10240) {
>   return 2048;
>   } else if (nmMemoryPerNodeNormalized <= 24576) {
>   return 3072;
>   } else {
>   return 4096;
>   }
> }
> 
> numConcurrentQueries, maxConcurrentQueries = (nmMemoryPerNodeNormalized, 
> nmCpusPerNode, memoryPerThread, numRequestedLlapNodes, 
> llapMemoryTezAMsAndDaemons, tezAmContainerSize, amCapacityAvailable) {
>   maxExecutorsPerNode = 

Re: Review Request 53883: 'HAWQ segments unregistered' shows incorrect alert

[Lav Jain]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53883/#review156325
---


Ship it!




Ship It!

- Lav Jain


On Nov. 18, 2016, 6:28 p.m., Matt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53883/
> ---
> 
> (Updated Nov. 18, 2016, 6:28 p.m.)
> 
> 
> Review request for Ambari and Lav Jain.
> 
> 
> Bugs: AMBARI-18935
> https://issues.apache.org/jira/browse/AMBARI-18935
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The alert is caused because of the mismatch in case between HAWQ segment 
> hostnames mentioned in the slaves file and the data stored in the DB
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/alerts/alert_segment_registration_status.py
>  ee8999e 
>   
> ambari-server/src/test/python/common-services/HAWQ/test_alert_segment_registration_status.py
>  f1fde066 
> 
> Diff: https://reviews.apache.org/r/53883/diff/
> 
> 
> Testing
> ---
> 
> Tested manually.
> 
> 
> Thanks,
> 
> Matt
> 
>

Review Request 53883: 'HAWQ segments unregistered' shows incorrect alert

[Matt]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53883/
---

Review request for Ambari and Lav Jain.


Bugs: AMBARI-18935
https://issues.apache.org/jira/browse/AMBARI-18935


Repository: ambari


Description
---

The alert is caused because of the mismatch in case between HAWQ segment 
hostnames mentioned in the slaves file and the data stored in the DB


Diffs
-

  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/alerts/alert_segment_registration_status.py
 ee8999e 
  
ambari-server/src/test/python/common-services/HAWQ/test_alert_segment_registration_status.py
 f1fde066 

Diff: https://reviews.apache.org/r/53883/diff/


Testing
---

Tested manually.


Thanks,

Matt

Re: Review Request 53854: Agent Auto Restart Doesn't Release Ping Port

[Dmytro Sen]

> On Ноя. 18, 2016, 1:08 п.п., Andrew Onischuk wrote:
> > ambari-agent/src/main/python/ambari_agent/AmbariAgent.py, line 59
> > 
> >
> > This will most likely kill child processes started by agent like 
> > datanode/namenode etc. I think they are in the same process group.
> > Can you please check?

Checked on Ubuntu and CentOS. DataNode, NameNode, etc. have init as a perent 
process with PPID 1. On ambari agent stop/restart they aren't killed


- Dmytro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53854/#review156296
---


On Ноя. 18, 2016, 1:05 п.п., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53854/
> ---
> 
> (Updated Ноя. 18, 2016, 1:05 п.п.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-18922
> https://issues.apache.org/jira/browse/AMBARI-18922
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ambari agent auto-restart fails
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/AmbariAgent.py d701e49 
>   ambari-agent/src/main/python/ambari_agent/StatusCommandsExecutor.py 5d3607a 
>   ambari-agent/src/test/python/ambari_agent/TestAmbariAgent.py 1be487c 
> 
> Diff: https://reviews.apache.org/r/53854/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 53854: Agent Auto Restart Doesn't Release Ping Port

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53854/#review156296
---


Fix it, then Ship it!





ambari-agent/src/main/python/ambari_agent/AmbariAgent.py (line 59)


This will most likely kill child processes started by agent like 
datanode/namenode etc. I think they are in the same process group.
Can you please check?


- Andrew Onischuk


On Nov. 18, 2016, 1:05 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53854/
> ---
> 
> (Updated Nov. 18, 2016, 1:05 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-18922
> https://issues.apache.org/jira/browse/AMBARI-18922
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Ambari agent auto-restart fails
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/AmbariAgent.py d701e49 
>   ambari-agent/src/main/python/ambari_agent/StatusCommandsExecutor.py 5d3607a 
>   ambari-agent/src/test/python/ambari_agent/TestAmbariAgent.py 1be487c 
> 
> Diff: https://reviews.apache.org/r/53854/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 53854: Agent Auto Restart Doesn't Release Ping Port

[Dmytro Sen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53854/
---

(Updated Ноя. 18, 2016, 1:05 п.п.)


Review request for Ambari, Andrew Onischuk, Jonathan Hurley, and Vitalyi 
Brodetskyi.


Bugs: AMBARI-18922
https://issues.apache.org/jira/browse/AMBARI-18922


Repository: ambari


Description
---

Ambari agent auto-restart fails


Diffs (updated)
-

  ambari-agent/src/main/python/ambari_agent/AmbariAgent.py d701e49 
  ambari-agent/src/main/python/ambari_agent/StatusCommandsExecutor.py 5d3607a 
  ambari-agent/src/test/python/ambari_agent/TestAmbariAgent.py 1be487c 

Diff: https://reviews.apache.org/r/53854/diff/


Testing
---

Unit tests passed


Thanks,

Dmytro Sen

Re: Review Request 53855: keytab settings in kerberos.json for SPARK are at the wrong place

[Robert Levas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53855/#review156290
---


Ship it!




Ship It!

- Robert Levas


On Nov. 18, 2016, 5:42 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53855/
> ---
> 
> (Updated Nov. 18, 2016, 5:42 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Laszlo Puskas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-18923
> https://issues.apache.org/jira/browse/AMBARI-18923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The livy user keytabs settings should be specified at LIVY_SERVER component 
> level instead of service level.
> This causes that all hosts that contain a SPARK component will get the keytab 
> file for livy. Only the host that has LIVY_SERVER should get this keytab.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json 
> 934f3c6 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> 0689e7f 
> 
> Diff: https://reviews.apache.org/r/53855/diff/
> 
> 
> Testing
> ---
> 
> only json confiugration changed, no new unittest was added
> 
> Manual testing. 
> 1. performed an express upgrade from HDP 2.4 -> 2.5
> 2. regenerted keytabs
> 3. checked there was no error
> 
> Running existing unittests (2 unrelated failure in the java tests):
> 
>  Python: Total run:1163 Total errors:0 Total failures:0
> 
>  Java: Tests run: 4772, Failures: 2, Errors: 0, Skipped: 37
>  Failed tests: 
>ServiceComponentHostTest.testStaleConfigs:858 null
>
> AsyncCallableServiceTest.testCallableServiceShouldCancelTaskWhenTimeoutExceeded:95
>  
>Unexpected method call ScheduledExecutorService.schedule(EasyMock for 
> interface java.util.concurrent.Callable, 500, MILLISECONDS):
> 
>  - ServiceComponentHostTest.testStaleConfigs:858 failed before applying the 
> patch as well
>  - Rerunning 
> AsyncCallableServiceTest.testCallableServiceShouldCancelTaskWhenTimeoutExceeded
>  passed at the second time
> 
> 
> File Attachments
> 
> 
> 2.4
>   
> https://reviews.apache.org/media/uploaded/files/2016/11/17/7a5bad98-d981-46bb-96e8-10cbf782af7a__AMBARI-18923_branch-2.4.patch
> 2.5
>   
> https://reviews.apache.org/media/uploaded/files/2016/11/17/8a063970-bc7e-428a-a499-ec9addc05344__AMBARI-18923_branch-2.5.patch
> trunk
>   
> https://reviews.apache.org/media/uploaded/files/2016/11/18/a3dd7d79-56f1-4fcd-8fae-3079d1e7b80b__AMBARI-18923_trunk.patch
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 53855: keytab settings in kerberos.json for SPARK are at the wrong place

[Attila Magyar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53855/
---

(Updated Nov. 18, 2016, 10:42 a.m.)


Review request for Ambari, Attila Doroszlai, Laszlo Puskas, and Sebastian 
Toader.


Bugs: AMBARI-18923
https://issues.apache.org/jira/browse/AMBARI-18923


Repository: ambari


Description
---

The livy user keytabs settings should be specified at LIVY_SERVER component 
level instead of service level.
This causes that all hosts that contain a SPARK component will get the keytab 
file for livy. Only the host that has LIVY_SERVER should get this keytab.


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json 
934f3c6 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
0689e7f 

Diff: https://reviews.apache.org/r/53855/diff/


Testing (updated)
---

only json confiugration changed, no new unittest was added

Manual testing. 
1. performed an express upgrade from HDP 2.4 -> 2.5
2. regenerted keytabs
3. checked there was no error

Running existing unittests (2 unrelated failure in the java tests):

 Python: Total run:1163 Total errors:0 Total failures:0

 Java: Tests run: 4772, Failures: 2, Errors: 0, Skipped: 37
 Failed tests: 
   ServiceComponentHostTest.testStaleConfigs:858 null
   
AsyncCallableServiceTest.testCallableServiceShouldCancelTaskWhenTimeoutExceeded:95
 
   Unexpected method call ScheduledExecutorService.schedule(EasyMock for 
interface java.util.concurrent.Callable, 500, MILLISECONDS):

 - ServiceComponentHostTest.testStaleConfigs:858 failed before applying the 
patch as well
 - Rerunning 
AsyncCallableServiceTest.testCallableServiceShouldCancelTaskWhenTimeoutExceeded 
passed at the second time


File Attachments


2.4
  
https://reviews.apache.org/media/uploaded/files/2016/11/17/7a5bad98-d981-46bb-96e8-10cbf782af7a__AMBARI-18923_branch-2.4.patch
2.5
  
https://reviews.apache.org/media/uploaded/files/2016/11/17/8a063970-bc7e-428a-a499-ec9addc05344__AMBARI-18923_branch-2.5.patch
trunk
  
https://reviews.apache.org/media/uploaded/files/2016/11/18/a3dd7d79-56f1-4fcd-8fae-3079d1e7b80b__AMBARI-18923_trunk.patch


Thanks,

Attila Magyar

Re: Review Request 53855: keytab settings in kerberos.json for SPARK are at the wrong place

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53855/#review156289
---


Ship it!




Ship It!

- Sebastian Toader


On Nov. 18, 2016, 10:06 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53855/
> ---
> 
> (Updated Nov. 18, 2016, 10:06 a.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Laszlo Puskas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-18923
> https://issues.apache.org/jira/browse/AMBARI-18923
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The livy user keytabs settings should be specified at LIVY_SERVER component 
> level instead of service level.
> This causes that all hosts that contain a SPARK component will get the keytab 
> file for livy. Only the host that has LIVY_SERVER should get this keytab.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json 
> 934f3c6 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> 0689e7f 
> 
> Diff: https://reviews.apache.org/r/53855/diff/
> 
> 
> Testing
> ---
> 
> only json confiugration changed, no unitttests
> 
> Manual testing. 
> 1. performed an express upgrade from HDP 2.4 -> 2.5
> 2. regenerted keytabs
> 3. checked there was no error
> 
> unittests are still running..
> 
> 
> File Attachments
> 
> 
> 2.4
>   
> https://reviews.apache.org/media/uploaded/files/2016/11/17/7a5bad98-d981-46bb-96e8-10cbf782af7a__AMBARI-18923_branch-2.4.patch
> 2.5
>   
> https://reviews.apache.org/media/uploaded/files/2016/11/17/8a063970-bc7e-428a-a499-ec9addc05344__AMBARI-18923_branch-2.5.patch
> trunk
>   
> https://reviews.apache.org/media/uploaded/files/2016/11/18/a3dd7d79-56f1-4fcd-8fae-3079d1e7b80b__AMBARI-18923_trunk.patch
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Review Request 53855: keytab settings in kerberos.json for SPARK are at the wrong place

[Attila Magyar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53855/
---

Review request for Ambari, Attila Doroszlai, Laszlo Puskas, and Sebastian 
Toader.


Bugs: AMBARI-18923
https://issues.apache.org/jira/browse/AMBARI-18923


Repository: ambari


Description
---

The livy user keytabs settings should be specified at LIVY_SERVER component 
level instead of service level.
This causes that all hosts that contain a SPARK component will get the keytab 
file for livy. Only the host that has LIVY_SERVER should get this keytab.


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json 
934f3c6 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
0689e7f 

Diff: https://reviews.apache.org/r/53855/diff/


Testing
---

only json confiugration changed, no unitttests

Manual testing. 
1. performed an express upgrade from HDP 2.4 -> 2.5
2. regenerted keytabs
3. checked there was no error

unittests are still running..


File Attachments


2.4
  
https://reviews.apache.org/media/uploaded/files/2016/11/17/7a5bad98-d981-46bb-96e8-10cbf782af7a__AMBARI-18923_branch-2.4.patch
2.5
  
https://reviews.apache.org/media/uploaded/files/2016/11/17/8a063970-bc7e-428a-a499-ec9addc05344__AMBARI-18923_branch-2.5.patch
trunk
  
https://reviews.apache.org/media/uploaded/files/2016/11/18/a3dd7d79-56f1-4fcd-8fae-3079d1e7b80b__AMBARI-18923_trunk.patch


Thanks,

Attila Magyar