Re: Review Request 61017: AMS fail to start after IOP 4.2 to HDP 2.6.2 upgrade

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61017/#review181088
---


Ship it!




Ship It!

- Jonathan Hurley


On July 20, 2017, 9:51 p.m., Sid Wagle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61017/
> ---
> 
> (Updated July 20, 2017, 9:51 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, Aravindan 
> Vijayan, Jonathan Hurley, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21542
> https://issues.apache.org/jira/browse/AMBARI-21542
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After IOP 4.2 to HDP 2.6.2 upgrade, AMS fails to start due to missing Grafana 
> configuration.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-grafana-env.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-grafana-ini.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-hbase-site.xml
>  0432ffa 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-site.xml
>  0a25a9f 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-ssl-client.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-ssl-server.xml
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61017/diff/2/
> 
> 
> Testing
> ---
> 
> Verfied manually by upgrading cluster with AMS and HDFS. And adding Grafana 
> successfully.
> 
> Ran python tests:
> 
> --
> Ran 274 tests in 7.116s
> 
> OK
> --
> Total run:1162
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Sid Wagle
> 
>

Re: Review Request 61017: AMS fail to start after IOP 4.2 to HDP 2.6.2 upgrade

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61017/
---

(Updated July 21, 2017, 1:50 a.m.)


Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, Jonathan 
Hurley, and Sumit Mohanty.


Changes
---

Reverting changes to shared initialization already committed to branch from the 
patch.


Bugs: AMBARI-21542
https://issues.apache.org/jira/browse/AMBARI-21542


Repository: ambari


Description
---

After IOP 4.2 to HDP 2.6.2 upgrade, AMS fails to start due to missing Grafana 
configuration.


Diffs (updated)
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-grafana-env.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-grafana-ini.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-hbase-site.xml
 0432ffa 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-site.xml
 0a25a9f 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-ssl-client.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-ssl-server.xml
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61017/diff/2/

Changes: https://reviews.apache.org/r/61017/diff/1-2/


Testing
---

Verfied manually by upgrading cluster with AMS and HDFS. And adding Grafana 
successfully.

Ran python tests:

--
Ran 274 tests in 7.116s

OK
--
Total run:1162
Total errors:0
Total failures:0
OK


Thanks,

Sid Wagle

Review Request 61017: AMS fail to start after IOP 4.2 to HDP 2.6.2 upgrade

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61017/
---

Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, Jonathan 
Hurley, and Sumit Mohanty.


Bugs: AMBARI-21542
https://issues.apache.org/jira/browse/AMBARI-21542


Repository: ambari


Description
---

After IOP 4.2 to HDP 2.6.2 upgrade, AMS fails to start due to missing Grafana 
configuration.


Diffs (updated)
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.0/hooks/before-ANY/scripts/shared_initialization.py
 1c98fb8 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-ANY/scripts/shared_initialization.py
 930ed1f 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-grafana-env.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-grafana-ini.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-hbase-site.xml
 0432ffa 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-site.xml
 0a25a9f 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-ssl-client.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/AMBARI_METRICS/configuration/ams-ssl-server.xml
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61017/diff/1/


Testing
---

Verfied manually by upgrading cluster with AMS and HDFS. And adding Grafana 
successfully.

Ran python tests:

--
Ran 274 tests in 7.116s

OK
--
Total run:1162
Total errors:0
Total failures:0
OK


Thanks,

Sid Wagle

Re: Review Request 61012: AMBARI-21463. Cross-stack upgrade, Oozie restart fails with ext-2.2.zip missing error

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61012/#review181084
---


Ship it!




Ship It!

- Sid Wagle


On July 20, 2017, 11:15 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61012/
> ---
> 
> (Updated July 20, 2017, 11:15 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Jonathan Hurley, Sumit 
> Mohanty, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21463
> https://issues.apache.org/jira/browse/AMBARI-21463
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Oozie Server restart failed due to this: Unable to copy ext-2.2.zip because 
> it does not exist
> during a cross-stack migration from BigInsights to HDP.
> 
> In BigInsights 4.2.5, the source path was 
> /usr/share/BIGINSIGHTS-oozie/ext-2.2.zip, but in BigInsights 4.2.0, the 
> source path was /var/lib/oozie/ext-2.2.zip
> So will append both to a list and copy the first one found.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
>  142e962 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
>  719fb32 
>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 
> 426c36a 
> 
> 
> Diff: https://reviews.apache.org/r/61012/diff/1/
> 
> 
> Testing
> ---
> 
> Python unit tests passed,
> 
> --
> Total run:1161
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 61012: AMBARI-21463. Cross-stack upgrade, Oozie restart fails with ext-2.2.zip missing error

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61012/#review181082
---




ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
Lines 287 (patched)


Earlier the code would log/fail if path did not exist. But now it will 
skip. So if there is no successful copy should we log that nothing got copied?


- Sumit Mohanty


On July 20, 2017, 11:15 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61012/
> ---
> 
> (Updated July 20, 2017, 11:15 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Jonathan Hurley, Sumit 
> Mohanty, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21463
> https://issues.apache.org/jira/browse/AMBARI-21463
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Oozie Server restart failed due to this: Unable to copy ext-2.2.zip because 
> it does not exist
> during a cross-stack migration from BigInsights to HDP.
> 
> In BigInsights 4.2.5, the source path was 
> /usr/share/BIGINSIGHTS-oozie/ext-2.2.zip, but in BigInsights 4.2.0, the 
> source path was /var/lib/oozie/ext-2.2.zip
> So will append both to a list and copy the first one found.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
>  142e962 
>   
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
>  719fb32 
>   ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 
> 426c36a 
> 
> 
> Diff: https://reviews.apache.org/r/61012/diff/1/
> 
> 
> Testing
> ---
> 
> Python unit tests passed,
> 
> --
> Total run:1161
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Review Request 61012: AMBARI-21463. Cross-stack upgrade, Oozie restart fails with ext-2.2.zip missing error

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61012/
---

Review request for Ambari, Dmytro Grinenko, Di Li, Jonathan Hurley, Sumit 
Mohanty, Sid Wagle, and Tim Thorpe.


Bugs: AMBARI-21463
https://issues.apache.org/jira/browse/AMBARI-21463


Repository: ambari


Description
---

Oozie Server restart failed due to this: Unable to copy ext-2.2.zip because it 
does not exist
during a cross-stack migration from BigInsights to HDP.

In BigInsights 4.2.5, the source path was 
/usr/share/BIGINSIGHTS-oozie/ext-2.2.zip, but in BigInsights 4.2.0, the source 
path was /var/lib/oozie/ext-2.2.zip
So will append both to a list and copy the first one found.


Diffs
-

  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py
 142e962 
  
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server_upgrade.py
 719fb32 
  ambari-server/src/test/python/stacks/2.0.6/OOZIE/test_oozie_server.py 426c36a 


Diff: https://reviews.apache.org/r/61012/diff/1/


Testing
---

Python unit tests passed,

--
Total run:1161
Total errors:0
Total failures:0
OK


Thanks,

Alejandro Fernandez

Re: Review Request 61006: AMBARI-21541 Restart services failed post Ambari Upgrade

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61006/#review181080
---


Ship it!




Ship It!

- Alejandro Fernandez


On July 20, 2017, 8:35 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61006/
> ---
> 
> (Updated July 20, 2017, 8:35 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21541
> https://issues.apache.org/jira/browse/AMBARI-21541
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Py API was updated in AMBARI-21531. Client component restart fails after 
> Ambari upgrade while running custom hook script on Suse 11. This causes the 
> before-ANY hook in BI 4.2 and 4.2.5 stack to fail to execute with error:
> resource_management.core.exceptions.InvalidArgument: User['hive'] Expected an 
> integer for uid received '1001'
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/hooks/before-ANY/scripts/shared_initialization.py
>  1c98fb8 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-ANY/scripts/shared_initialization.py
>  930ed1f 
> 
> 
> Diff: https://reviews.apache.org/r/61006/diff/1/
> 
> 
> Testing
> ---
> 
> patch cluster, restart services.
> 
> 
> Thanks,
> 
> Di Li
> 
>

Review Request 61006: AMBARI-21541 Restart services failed post Ambari Upgrade

[Di Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61006/
---

Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan 
Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.


Bugs: AMBARI-21541
https://issues.apache.org/jira/browse/AMBARI-21541


Repository: ambari


Description
---

Py API was updated in AMBARI-21531. Client component restart fails after Ambari 
upgrade while running custom hook script on Suse 11. This causes the before-ANY 
hook in BI 4.2 and 4.2.5 stack to fail to execute with error:
resource_management.core.exceptions.InvalidArgument: User['hive'] Expected an 
integer for uid received '1001'


Diffs
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.0/hooks/before-ANY/scripts/shared_initialization.py
 1c98fb8 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-ANY/scripts/shared_initialization.py
 930ed1f 


Diff: https://reviews.apache.org/r/61006/diff/1/


Testing
---

patch cluster, restart services.


Thanks,

Di Li

Re: Review Request 60986: AMBARI-21528. Zookeeper server has incorrect memory setting, missing m in Xmx value

[Alejandro Fernandez]

> On July 20, 2017, 2:14 p.m., Attila Doroszlai wrote:
> > Normally the property is added during Ambari upgrade: initially with 
> > default value of "1024", then updated to "1024m" by `UpgradeCatalog222`.  
> > (Try upgrading from Apache Ambari 2.2.1 to 2.5.2.)
> > 
> > The root cause of the problem is that `zk_server_heapsize` is referenced in 
> > `zookeeper-env` (the `content`) in BigInsights 4.2, but the property itself 
> > is missing.  It is then added during stack upgrade with its raw default 
> > value.
> > 
> > I think the proper fix is to add the missing property in the BI 4.2 stack 
> > definition.  The current patch would be a nice workaround if there already 
> > were clusters with the broken value.
> 
> Jonathan Hurley wrote:
> I think that there are clusters with the broken value today.
> 
> Jonathan Hurley wrote:
> Ah, I see what you're saying. So, if we added it to the BI stack, then it 
> would get taken care of up Ambari Server upgrade automatically. We should do 
> that.
> 
> Di Li wrote:
> BI has it defined slightly differently though, we have it hardcoded as 
> zk_server_heapsize = "-Xmx1024m" in Zookeeper (in BI 4.2 and 4.2.5 stack) 
> params.py file.

Regardless of what the value in the config is, the python script will format it 
to be correct. New cluster installs have 1024, and migrated clusters will 
preserve their existing config value but have the python script fix it.


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60986/#review181042
---


On July 20, 2017, 12:13 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60986/
> ---
> 
> (Updated July 20, 2017, 12:13 a.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, 
> and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21528
> https://issues.apache.org/jira/browse/AMBARI-21528
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Repro Steps:
> 
> * Installed BI 4.2.0 cluster on IBM Ambari 2.2.2 with Zookeeper
> * Upgraded Ambari to 2.5.2.0-146
> * Registered HDP 2.6.2.0 repo, installed packages
> * Ran service checks
> * Started Express Upgrade
> 
> Result: _Service Check ZooKeeper_ step failed with {{KeeperErrorCode = 
> ConnectionLoss for /zk_smoketest}}
> 
> This was caused by Zookeeper dying immediately during restart:
> ```
> Error occurred during initialization of VM
> Too small initial heap
> ```
> 
> Before EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/iop/current/zookeeper-server
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024m
> export JAVA=$JAVA_HOME/bin/java
> export CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*
> ```
> 
> After EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-client
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024
> export JAVA=$JAVA_HOME/bin/java
> ```
> 
> Note missing "m" in memory setting.
> 
> zookeeper-env template contains,
> ```
> export SERVER_JVMFLAGS={{zk_server_heapsize}}
> ```
> 
> In this cluster, zookeeper-env contains,
> zk_server_heapsize: "1024"
> 
> While the params_linux.py file has some inconsistencies with appending the 
> letter "m".
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024m"))
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> Instead, it should be,
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024"))
> zk_server_heapsize_value = zk_server_heapsize_value.strip()
> if len(zk_server_heapsize_value) > 0 and not 
> zk_server_heapsize_value[-1].isdigit():
>   zk_server_heapsize_value = zk_server_heapsize_value + "m"
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py
>  0780d2e 
> 
> 
> Diff: https://reviews.apache.org/r/60986/diff/2/
> 
> 
> Testing
> ---
> 
> Python unit tests passed,
> 
> --
> Total run:1161
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 60999: Add host doesn't fully add a node when include/exclude files are used

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60999/#review181074
---


Fix it, then Ship it!





ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
Lines 3367 (patched)


Can you please add comments why we need to track this separately?



ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
Lines 3661 (patched)


Can we call this something more meaningful than temp2 ?


- Sid Wagle


On July 20, 2017, 2:49 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60999/
> ---
> 
> (Updated July 20, 2017, 2:49 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, Sid Wagle, and 
> Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21345
> https://issues.apache.org/jira/browse/AMBARI-21345
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Additional patch to solve naming issues
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  5180870 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  4229d34 
> 
> 
> Diff: https://reviews.apache.org/r/60999/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing.
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 61000: AMBARI-21527 Restart of MR2 History Server failed due to wrong NameNode RPC address

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61000/#review181071
---


Ship it!




Ship It!

- Alejandro Fernandez


On July 20, 2017, 4:54 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61000/
> ---
> 
> (Updated July 20, 2017, 4:54 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, Jonathan 
> Hurley, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21527
> https://issues.apache.org/jira/browse/AMBARI-21527
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> dfs.namenode.rpc-address gets merged in during EU with default value 
> "localhost:8020". Both the host and port could be wrong and cause issues 
> during HDFS restart. NN restart for kerberos cluster and remote DN restart 
> for non secured cluster. 
> 
> This is because HDFS Py script looks for that property first and uses it if 
> it exists. This logic and the fact that the property (seems unnecessarily) 
> merged in during EU with "localhost:8020" as the value was what caused the 
> issues in our tests.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  e476d57 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cbd0550 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  dada6e2 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  3ea20ed 
> 
> 
> Diff: https://reviews.apache.org/r/61000/diff/1/
> 
> 
> Testing
> ---
> 
> tested as part of the EU
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 60995: Namenode restart - PID file delete happens before the call to check status

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60995/#review181067
---


Ship it!




Ship It!

- Alejandro Fernandez


On July 20, 2017, 10:37 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60995/
> ---
> 
> (Updated July 20, 2017, 10:37 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21532
> https://issues.apache.org/jira/browse/AMBARI-21532
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> PID file delete happens before the call to check status.
> 
> {code}
> ...
> 2017-07-06 00:03:21,004 - 
> File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete']}
> 2017-07-06 00:05:21,103 - Waiting for actual component stop
> 2017-07-06 00:05:21,104 - Pid file 
> /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid is empty or does not exist
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/check_process_status.py
>  7961f008ca 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 2b374c5cff 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py
>  d861ba93a5 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/utils.py
>  53774c661a 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/utils.py
>  7dcbca8d5b 
> 
> 
> Diff: https://reviews.apache.org/r/60995/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 61000: AMBARI-21527 Restart of MR2 History Server failed due to wrong NameNode RPC address

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61000/#review181061
---


Ship it!




Ship It!

- Sid Wagle


On July 20, 2017, 4:54 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61000/
> ---
> 
> (Updated July 20, 2017, 4:54 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, Jonathan 
> Hurley, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21527
> https://issues.apache.org/jira/browse/AMBARI-21527
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> dfs.namenode.rpc-address gets merged in during EU with default value 
> "localhost:8020". Both the host and port could be wrong and cause issues 
> during HDFS restart. NN restart for kerberos cluster and remote DN restart 
> for non secured cluster. 
> 
> This is because HDFS Py script looks for that property first and uses it if 
> it exists. This logic and the fact that the property (seems unnecessarily) 
> merged in during EU with "localhost:8020" as the value was what caused the 
> issues in our tests.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  e476d57 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cbd0550 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  dada6e2 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  3ea20ed 
> 
> 
> Diff: https://reviews.apache.org/r/61000/diff/1/
> 
> 
> Testing
> ---
> 
> tested as part of the EU
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 61000: AMBARI-21527 Restart of MR2 History Server failed due to wrong NameNode RPC address

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61000/#review181059
---


Ship it!




Ship It!

- Tim Thorpe


On July 20, 2017, 4:54 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61000/
> ---
> 
> (Updated July 20, 2017, 4:54 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, Jonathan 
> Hurley, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21527
> https://issues.apache.org/jira/browse/AMBARI-21527
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> dfs.namenode.rpc-address gets merged in during EU with default value 
> "localhost:8020". Both the host and port could be wrong and cause issues 
> during HDFS restart. NN restart for kerberos cluster and remote DN restart 
> for non secured cluster. 
> 
> This is because HDFS Py script looks for that property first and uses it if 
> it exists. This logic and the fact that the property (seems unnecessarily) 
> merged in during EU with "localhost:8020" as the value was what caused the 
> issues in our tests.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  e476d57 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cbd0550 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  dada6e2 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  3ea20ed 
> 
> 
> Diff: https://reviews.apache.org/r/61000/diff/1/
> 
> 
> Testing
> ---
> 
> tested as part of the EU
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 61000: AMBARI-21527 Restart of MR2 History Server failed due to wrong NameNode RPC address

[Di Li]

> On July 20, 2017, 4:57 p.m., Sid Wagle wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
> > Lines 34 (patched)
> > 
> >
> > What if user has customized the port setting for rpc address different 
> > from 8020?

The point is to not use "dfs.namenode.rpc-address" as it has wrong value and 
causes NameNode Py script to finish calculation prematurely. So the fix is to 
only use fs.defaultFS for non-HA case. fs.defaultFS still has both FQDN and 
port and NameNode Py script has logic to handle it.


- Di


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61000/#review181055
---


On July 20, 2017, 4:54 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61000/
> ---
> 
> (Updated July 20, 2017, 4:54 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, Jonathan 
> Hurley, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21527
> https://issues.apache.org/jira/browse/AMBARI-21527
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> dfs.namenode.rpc-address gets merged in during EU with default value 
> "localhost:8020". Both the host and port could be wrong and cause issues 
> during HDFS restart. NN restart for kerberos cluster and remote DN restart 
> for non secured cluster. 
> 
> This is because HDFS Py script looks for that property first and uses it if 
> it exists. This logic and the fact that the property (seems unnecessarily) 
> merged in during EU with "localhost:8020" as the value was what caused the 
> issues in our tests.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  e476d57 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cbd0550 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  dada6e2 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  3ea20ed 
> 
> 
> Diff: https://reviews.apache.org/r/61000/diff/1/
> 
> 
> Testing
> ---
> 
> tested as part of the EU
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 61000: AMBARI-21527 Restart of MR2 History Server failed due to wrong NameNode RPC address

[Sid Wagle]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61000/#review181055
---




ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
Lines 34 (patched)


What if user has customized the port setting for rpc address different from 
8020?


- Sid Wagle


On July 20, 2017, 4:54 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61000/
> ---
> 
> (Updated July 20, 2017, 4:54 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, Jonathan 
> Hurley, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21527
> https://issues.apache.org/jira/browse/AMBARI-21527
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> dfs.namenode.rpc-address gets merged in during EU with default value 
> "localhost:8020". Both the host and port could be wrong and cause issues 
> during HDFS restart. NN restart for kerberos cluster and remote DN restart 
> for non secured cluster. 
> 
> This is because HDFS Py script looks for that property first and uses it if 
> it exists. This logic and the fact that the property (seems unnecessarily) 
> merged in during EU with "localhost:8020" as the value was what caused the 
> issues in our tests.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  e476d57 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  cbd0550 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  dada6e2 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  3ea20ed 
> 
> 
> Diff: https://reviews.apache.org/r/61000/diff/1/
> 
> 
> Testing
> ---
> 
> tested as part of the EU
> 
> 
> Thanks,
> 
> Di Li
> 
>

Review Request 61000: AMBARI-21527 Restart of MR2 History Server failed due to wrong NameNode RPC address

[Di Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61000/
---

Review request for Ambari, Attila Doroszlai, Alejandro Fernandez, Jonathan 
Hurley, and Tim Thorpe.


Bugs: AMBARI-21527
https://issues.apache.org/jira/browse/AMBARI-21527


Repository: ambari


Description
---

dfs.namenode.rpc-address gets merged in during EU with default value 
"localhost:8020". Both the host and port could be wrong and cause issues during 
HDFS restart. NN restart for kerberos cluster and remote DN restart for non 
secured cluster. 

This is because HDFS Py script looks for that property first and uses it if it 
exists. This logic and the fact that the property (seems unnecessarily) merged 
in during EU with "localhost:8020" as the value was what caused the issues in 
our tests.


Diffs
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
 e476d57 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 cbd0550 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
 dada6e2 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 3ea20ed 


Diff: https://reviews.apache.org/r/61000/diff/1/


Testing
---

tested as part of the EU


Thanks,

Di Li

Re: Review Request 60861: Ambari rebuilds custom auth_to_local rules changing its case sensitiveness option (/L)

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60861/#review181054
---


Ship it!




Ship It!

- Sebastian Toader


On July 17, 2017, 1:44 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60861/
> ---
> 
> (Updated July 17, 2017, 1:44 p.m.)
> 
> 
> Review request for Ambari, Laszlo Puskas, Robert Levas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21418
> https://issues.apache.org/jira/browse/AMBARI-21418
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When auth to local rules are rebuilt (add new service to kerberized cluster, 
> regenerate keytab, enable kerberos) ambari strips down the /L suffix from the 
> rules that were added by the user.
> 
> The desired behaviour is the following:
>  1. Ambari shouldn't touch the custom rules that were added by the user
>  2. Ambari is allowed to regenerate and modify the ambari managed rule based 
> on the case_insensitive_username_rules
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AuthToLocalBuilder.java
>  1d4abdd 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AuthToLocalBuilderTest.java
>  c08247d 
> 
> 
> Diff: https://reviews.apache.org/r/60861/diff/1/
> 
> 
> Testing
> ---
> 
> Tested the following scenarios:
> 1.
>  - added a custom rule with /L flag: 
> RULE:\[1:$1@$0\]\(.*@HDP01.LOCAL\)s/.\*/ambari-qa//L
>  - rebuilt auth to local rules
>  - check that the custom rule is still there and it still has the /L suffix, 
> and no other rules are changed
> 2.
>  - set case_insensitive_username_rules = false
>  - added a /L suffix to an ambari managed rule
>  - rebuilt auth to local rules
>  - checked that the ambari managed rule has no /L suffix
> 3.
>  - set case_insensitive_username_rules = true
>  - removed the /L suffix from the ambari managed rule
>  - rebuilt auth to local rules
>  - checked that the ambari managed rule has the /L suffix
> 4.
>  - set case_insensitive_username_rules = false
>  - rebuilt auth to local rules
>  - set case_insensitive_username_rules = true
>  - rebuilt auth to local rules
>  - checked that /L suffix was added to the ambari managed rule
> 
> existing tests: 
> 
> Results :
> Tests run: 4813, Failures: 0, Errors: 0, Skipped: 35
> --
> Total run:1145
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 60986: AMBARI-21528. Zookeeper server has incorrect memory setting, missing m in Xmx value

[Di Li]

> On July 20, 2017, 2:14 p.m., Attila Doroszlai wrote:
> > Normally the property is added during Ambari upgrade: initially with 
> > default value of "1024", then updated to "1024m" by `UpgradeCatalog222`.  
> > (Try upgrading from Apache Ambari 2.2.1 to 2.5.2.)
> > 
> > The root cause of the problem is that `zk_server_heapsize` is referenced in 
> > `zookeeper-env` (the `content`) in BigInsights 4.2, but the property itself 
> > is missing.  It is then added during stack upgrade with its raw default 
> > value.
> > 
> > I think the proper fix is to add the missing property in the BI 4.2 stack 
> > definition.  The current patch would be a nice workaround if there already 
> > were clusters with the broken value.
> 
> Jonathan Hurley wrote:
> I think that there are clusters with the broken value today.
> 
> Jonathan Hurley wrote:
> Ah, I see what you're saying. So, if we added it to the BI stack, then it 
> would get taken care of up Ambari Server upgrade automatically. We should do 
> that.

BI has it defined slightly differently though, we have it hardcoded as 
zk_server_heapsize = "-Xmx1024m" in Zookeeper (in BI 4.2 and 4.2.5 stack) 
params.py file.


- Di


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60986/#review181042
---


On July 20, 2017, 12:13 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60986/
> ---
> 
> (Updated July 20, 2017, 12:13 a.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, 
> and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21528
> https://issues.apache.org/jira/browse/AMBARI-21528
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Repro Steps:
> 
> * Installed BI 4.2.0 cluster on IBM Ambari 2.2.2 with Zookeeper
> * Upgraded Ambari to 2.5.2.0-146
> * Registered HDP 2.6.2.0 repo, installed packages
> * Ran service checks
> * Started Express Upgrade
> 
> Result: _Service Check ZooKeeper_ step failed with {{KeeperErrorCode = 
> ConnectionLoss for /zk_smoketest}}
> 
> This was caused by Zookeeper dying immediately during restart:
> ```
> Error occurred during initialization of VM
> Too small initial heap
> ```
> 
> Before EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/iop/current/zookeeper-server
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024m
> export JAVA=$JAVA_HOME/bin/java
> export CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*
> ```
> 
> After EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-client
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024
> export JAVA=$JAVA_HOME/bin/java
> ```
> 
> Note missing "m" in memory setting.
> 
> zookeeper-env template contains,
> ```
> export SERVER_JVMFLAGS={{zk_server_heapsize}}
> ```
> 
> In this cluster, zookeeper-env contains,
> zk_server_heapsize: "1024"
> 
> While the params_linux.py file has some inconsistencies with appending the 
> letter "m".
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024m"))
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> Instead, it should be,
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024"))
> zk_server_heapsize_value = zk_server_heapsize_value.strip()
> if len(zk_server_heapsize_value) > 0 and not 
> zk_server_heapsize_value[-1].isdigit():
>   zk_server_heapsize_value = zk_server_heapsize_value + "m"
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py
>  0780d2e 
> 
> 
> Diff: https://reviews.apache.org/r/60986/diff/2/
> 
> 
> Testing
> ---
> 
> Python unit tests passed,
> 
> --
> Total run:1161
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Review Request 60999: Add host doesn't fully add a node when include/exclude files are used

[Dmytro Sen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60999/
---

Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, Sid Wagle, and 
Vitalyi Brodetskyi.


Bugs: AMBARI-21345
https://issues.apache.org/jira/browse/AMBARI-21345


Repository: ambari


Description
---

Additional patch to solve naming issues


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 5180870 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 4229d34 


Diff: https://reviews.apache.org/r/60999/diff/1/


Testing
---

Manual testing.


Thanks,

Dmytro Sen

Re: Review Request 60986: AMBARI-21528. Zookeeper server has incorrect memory setting, missing m in Xmx value

[Jonathan Hurley]

> On July 20, 2017, 10:14 a.m., Attila Doroszlai wrote:
> > Normally the property is added during Ambari upgrade: initially with 
> > default value of "1024", then updated to "1024m" by `UpgradeCatalog222`.  
> > (Try upgrading from Apache Ambari 2.2.1 to 2.5.2.)
> > 
> > The root cause of the problem is that `zk_server_heapsize` is referenced in 
> > `zookeeper-env` (the `content`) in BigInsights 4.2, but the property itself 
> > is missing.  It is then added during stack upgrade with its raw default 
> > value.
> > 
> > I think the proper fix is to add the missing property in the BI 4.2 stack 
> > definition.  The current patch would be a nice workaround if there already 
> > were clusters with the broken value.
> 
> Jonathan Hurley wrote:
> I think that there are clusters with the broken value today.

Ah, I see what you're saying. So, if we added it to the BI stack, then it would 
get taken care of up Ambari Server upgrade automatically. We should do that.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60986/#review181042
---


On July 19, 2017, 8:13 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60986/
> ---
> 
> (Updated July 19, 2017, 8:13 p.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, 
> and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21528
> https://issues.apache.org/jira/browse/AMBARI-21528
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Repro Steps:
> 
> * Installed BI 4.2.0 cluster on IBM Ambari 2.2.2 with Zookeeper
> * Upgraded Ambari to 2.5.2.0-146
> * Registered HDP 2.6.2.0 repo, installed packages
> * Ran service checks
> * Started Express Upgrade
> 
> Result: _Service Check ZooKeeper_ step failed with {{KeeperErrorCode = 
> ConnectionLoss for /zk_smoketest}}
> 
> This was caused by Zookeeper dying immediately during restart:
> ```
> Error occurred during initialization of VM
> Too small initial heap
> ```
> 
> Before EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/iop/current/zookeeper-server
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024m
> export JAVA=$JAVA_HOME/bin/java
> export CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*
> ```
> 
> After EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-client
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024
> export JAVA=$JAVA_HOME/bin/java
> ```
> 
> Note missing "m" in memory setting.
> 
> zookeeper-env template contains,
> ```
> export SERVER_JVMFLAGS={{zk_server_heapsize}}
> ```
> 
> In this cluster, zookeeper-env contains,
> zk_server_heapsize: "1024"
> 
> While the params_linux.py file has some inconsistencies with appending the 
> letter "m".
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024m"))
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> Instead, it should be,
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024"))
> zk_server_heapsize_value = zk_server_heapsize_value.strip()
> if len(zk_server_heapsize_value) > 0 and not 
> zk_server_heapsize_value[-1].isdigit():
>   zk_server_heapsize_value = zk_server_heapsize_value + "m"
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py
>  0780d2e 
> 
> 
> Diff: https://reviews.apache.org/r/60986/diff/2/
> 
> 
> Testing
> ---
> 
> Python unit tests passed,
> 
> --
> Total run:1161
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 60986: AMBARI-21528. Zookeeper server has incorrect memory setting, missing m in Xmx value

[Jonathan Hurley]

> On July 20, 2017, 10:14 a.m., Attila Doroszlai wrote:
> > Normally the property is added during Ambari upgrade: initially with 
> > default value of "1024", then updated to "1024m" by `UpgradeCatalog222`.  
> > (Try upgrading from Apache Ambari 2.2.1 to 2.5.2.)
> > 
> > The root cause of the problem is that `zk_server_heapsize` is referenced in 
> > `zookeeper-env` (the `content`) in BigInsights 4.2, but the property itself 
> > is missing.  It is then added during stack upgrade with its raw default 
> > value.
> > 
> > I think the proper fix is to add the missing property in the BI 4.2 stack 
> > definition.  The current patch would be a nice workaround if there already 
> > were clusters with the broken value.

I think that there are clusters with the broken value today.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60986/#review181042
---


On July 19, 2017, 8:13 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60986/
> ---
> 
> (Updated July 19, 2017, 8:13 p.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, 
> and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21528
> https://issues.apache.org/jira/browse/AMBARI-21528
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Repro Steps:
> 
> * Installed BI 4.2.0 cluster on IBM Ambari 2.2.2 with Zookeeper
> * Upgraded Ambari to 2.5.2.0-146
> * Registered HDP 2.6.2.0 repo, installed packages
> * Ran service checks
> * Started Express Upgrade
> 
> Result: _Service Check ZooKeeper_ step failed with {{KeeperErrorCode = 
> ConnectionLoss for /zk_smoketest}}
> 
> This was caused by Zookeeper dying immediately during restart:
> ```
> Error occurred during initialization of VM
> Too small initial heap
> ```
> 
> Before EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/iop/current/zookeeper-server
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024m
> export JAVA=$JAVA_HOME/bin/java
> export CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*
> ```
> 
> After EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-client
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024
> export JAVA=$JAVA_HOME/bin/java
> ```
> 
> Note missing "m" in memory setting.
> 
> zookeeper-env template contains,
> ```
> export SERVER_JVMFLAGS={{zk_server_heapsize}}
> ```
> 
> In this cluster, zookeeper-env contains,
> zk_server_heapsize: "1024"
> 
> While the params_linux.py file has some inconsistencies with appending the 
> letter "m".
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024m"))
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> Instead, it should be,
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024"))
> zk_server_heapsize_value = zk_server_heapsize_value.strip()
> if len(zk_server_heapsize_value) > 0 and not 
> zk_server_heapsize_value[-1].isdigit():
>   zk_server_heapsize_value = zk_server_heapsize_value + "m"
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py
>  0780d2e 
> 
> 
> Diff: https://reviews.apache.org/r/60986/diff/2/
> 
> 
> Testing
> ---
> 
> Python unit tests passed,
> 
> --
> Total run:1161
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 60986: AMBARI-21528. Zookeeper server has incorrect memory setting, missing m in Xmx value

[Attila Doroszlai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60986/#review181042
---



Normally the property is added during Ambari upgrade: initially with default 
value of "1024", then updated to "1024m" by `UpgradeCatalog222`.  (Try 
upgrading from Apache Ambari 2.2.1 to 2.5.2.)

The root cause of the problem is that `zk_server_heapsize` is referenced in 
`zookeeper-env` (the `content`) in BigInsights 4.2, but the property itself is 
missing.  It is then added during stack upgrade with its raw default value.

I think the proper fix is to add the missing property in the BI 4.2 stack 
definition.  The current patch would be a nice workaround if there already were 
clusters with the broken value.

- Attila Doroszlai


On July 20, 2017, 2:13 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60986/
> ---
> 
> (Updated July 20, 2017, 2:13 a.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, 
> and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21528
> https://issues.apache.org/jira/browse/AMBARI-21528
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Repro Steps:
> 
> * Installed BI 4.2.0 cluster on IBM Ambari 2.2.2 with Zookeeper
> * Upgraded Ambari to 2.5.2.0-146
> * Registered HDP 2.6.2.0 repo, installed packages
> * Ran service checks
> * Started Express Upgrade
> 
> Result: _Service Check ZooKeeper_ step failed with {{KeeperErrorCode = 
> ConnectionLoss for /zk_smoketest}}
> 
> This was caused by Zookeeper dying immediately during restart:
> ```
> Error occurred during initialization of VM
> Too small initial heap
> ```
> 
> Before EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/iop/current/zookeeper-server
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024m
> export JAVA=$JAVA_HOME/bin/java
> export CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*
> ```
> 
> After EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-client
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024
> export JAVA=$JAVA_HOME/bin/java
> ```
> 
> Note missing "m" in memory setting.
> 
> zookeeper-env template contains,
> ```
> export SERVER_JVMFLAGS={{zk_server_heapsize}}
> ```
> 
> In this cluster, zookeeper-env contains,
> zk_server_heapsize: "1024"
> 
> While the params_linux.py file has some inconsistencies with appending the 
> letter "m".
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024m"))
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> Instead, it should be,
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024"))
> zk_server_heapsize_value = zk_server_heapsize_value.strip()
> if len(zk_server_heapsize_value) > 0 and not 
> zk_server_heapsize_value[-1].isdigit():
>   zk_server_heapsize_value = zk_server_heapsize_value + "m"
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py
>  0780d2e 
> 
> 
> Diff: https://reviews.apache.org/r/60986/diff/2/
> 
> 
> Testing
> ---
> 
> Python unit tests passed,
> 
> --
> Total run:1161
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 60943: Log Search Solr output properties should be provided by the Config API

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60943/#review181041
---


Ship it!




Ship It!

- Oliver Szabo


On July 20, 2017, 9:54 a.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60943/
> ---
> 
> (Updated July 20, 2017, 9:54 a.m.)
> 
> 
> Review request for Ambari, Oliver Szabo and Robert Nettleton.
> 
> 
> Bugs: AMBARI-21507
> https://issues.apache.org/jira/browse/AMBARI-21507
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The Solr output properties should be persisted into the Config API by the Log 
> Search Server as the output is initialized, and fetched by he Log Feeders 
> from there
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfig.java
>  6c5cefd 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/OutputConfigMonitor.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/outputconfig/OutputProperties.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/outputconfig/OutputSolrProperties.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass1.java
>  28844d5 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass2.java
>  5934fa6 
>   
> ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigZK.java
>  fdd8ed6 
>   
> ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/outputconfig/impl/OutputSolrPropertiesImpl.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  59c2a22 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
>  5bf074c 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
>  5356159 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
>  09fc3f5 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java
>  f1002ae 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/Output.java
>  65b9e19 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
>  4d6c43b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
>  162a7f8 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java
>  8985110 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrAuditLogPropsConfig.java
>  c569a27 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrEventHistoryPropsConfig.java
>  975e6a7 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrPropsConfig.java
>  ceddf7e 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrServiceLogPropsConfig.java
>  e5039d5 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/LogSearchConfigConfigurer.java
>  c34dce6 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/SolrCollectionConfigurer.java
>  f2d022e 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/AuditSolrDao.java
>  3eea08f 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/ServiceLogsSolrDao.java
>  308ef1f 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/SolrDaoBase.java
>  b30b6ef 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/handler/CreateCollectionHandler.java
>  752a1e1 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/ShipperConfigManager.java
>  2c143c0 
>   
> ambari-logsearch/docker/test-config/logfeeder/shipper-conf/output.config.json 
> 55fd36c 
>   
> am

Re: Review Request 60995: Namenode restart - PID file delete happens before the call to check status

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60995/#review181040
---


Ship it!




Ship It!

- Jonathan Hurley


On July 20, 2017, 6:37 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60995/
> ---
> 
> (Updated July 20, 2017, 6:37 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21532
> https://issues.apache.org/jira/browse/AMBARI-21532
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> PID file delete happens before the call to check status.
> 
> {code}
> ...
> 2017-07-06 00:03:21,004 - 
> File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete']}
> 2017-07-06 00:05:21,103 - Waiting for actual component stop
> 2017-07-06 00:05:21,104 - Pid file 
> /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid is empty or does not exist
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/check_process_status.py
>  7961f008ca 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 2b374c5cff 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py
>  d861ba93a5 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/utils.py
>  53774c661a 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/utils.py
>  7dcbca8d5b 
> 
> 
> Diff: https://reviews.apache.org/r/60995/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 60995: Namenode restart - PID file delete happens before the call to check status

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60995/#review181039
---




ambari-common/src/main/python/resource_management/libraries/functions/check_process_status.py
Lines 74-77 (patched)


Do we have a reusable timer class which repeats? This might be a good time 
to write one. Nothing wrong with the sleep/while loop, but we might be able to 
write something more reusable. Your call.

If we keep this, then is 0.1s too aggresive? Could we make it 1s instead?


- Jonathan Hurley


On July 20, 2017, 6:37 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60995/
> ---
> 
> (Updated July 20, 2017, 6:37 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21532
> https://issues.apache.org/jira/browse/AMBARI-21532
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> PID file delete happens before the call to check status.
> 
> {code}
> ...
> 2017-07-06 00:03:21,004 - 
> File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] {'action': ['delete']}
> 2017-07-06 00:05:21,103 - Waiting for actual component stop
> 2017-07-06 00:05:21,104 - Pid file 
> /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid is empty or does not exist
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/check_process_status.py
>  7961f008ca 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 2b374c5cff 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py
>  d861ba93a5 
>   
> ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/utils.py
>  53774c661a 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/utils.py
>  7dcbca8d5b 
> 
> 
> Diff: https://reviews.apache.org/r/60995/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

[Dmitro Lisnichenko]

> On July 20, 2017, 4:29 p.m., Jonathan Hurley wrote:
> > Ping. I think this review was submitted ... can it be closed?

pong. You are right, closing


- Dmitro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/#review181037
---


On June 29, 2017, 7:28 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60543/
> ---
> 
> (Updated June 29, 2017, 7:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21364
> https://issues.apache.org/jira/browse/AMBARI-21364
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129
> # Upgrade Ambari to 2.5.2.0-74 (hash: 
> fd30644590991deb41241454d6e9091ed7a38e92)
> # Run "ambari-server start"
> 
> {code}
> root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
> restart
> Using python  /usr/bin/python
> Restarting ambari-server
> Waiting for server stop...
> Ambari Server stopped
> Ambari Server running with administrator privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Ambari database consistency check started...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Waiting for server start..
> DB configs consistency check failed. Run "ambari-server start 
> --skip-database-check" to skip. You may try --auto-fix-database flag to 
> attempt to fix issues automatically. If you use this "--skip-database-check" 
> option, do not make any changes to your cluster topology or perform a cluster 
> upgrade until you correct the database consistency issues. See 
> /var/log/ambari-server/ambari-server-check-database.log for more details on 
> the consistency issues.
> ERROR: Exiting with exit code -1.
> REASON: Ambari Server java process has stopped. Please check the logs for 
> more information.
> {code}
> 
> DB log: ambari-server-check-database.log
> {code}
> 2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
> 2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
> configs that we got from db
> 2017-06-27 13:51:38,748  INFO - Getting services from metainfo
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KNOX
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KERBEROS
> 2017-06-27 13:51:38,749  INFO - Comparing required service configs from stack 
> with mapped service configs from db
> 2017-06-27 13:51:38,751 ERROR - Required config(s): product-info is(are) not 
> available for service SMARTSENSE with service config version 2 in cluster cl1
> 2017-06-27 13:51:38,751 ERROR - Required config(s): parquet-logging is(are) 
> not available for service HIVE with service config version 7 in cluster cl1
> 2017-06-27 13:51:38,751  INFO - Getting services which has mapped configs 
> which are not selected in clusterconfigmapping
> 2017-06-27 13:51:38,755  INFO - Checking Topology tables
> 2017-06-27 13:51:38,757  INFO -

Re: Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/#review181037
---



Ping. I think this review was submitted ... can it be closed?

- Jonathan Hurley


On June 29, 2017, 12:28 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60543/
> ---
> 
> (Updated June 29, 2017, 12:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21364
> https://issues.apache.org/jira/browse/AMBARI-21364
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129
> # Upgrade Ambari to 2.5.2.0-74 (hash: 
> fd30644590991deb41241454d6e9091ed7a38e92)
> # Run "ambari-server start"
> 
> {code}
> root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
> restart
> Using python  /usr/bin/python
> Restarting ambari-server
> Waiting for server stop...
> Ambari Server stopped
> Ambari Server running with administrator privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Ambari database consistency check started...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Waiting for server start..
> DB configs consistency check failed. Run "ambari-server start 
> --skip-database-check" to skip. You may try --auto-fix-database flag to 
> attempt to fix issues automatically. If you use this "--skip-database-check" 
> option, do not make any changes to your cluster topology or perform a cluster 
> upgrade until you correct the database consistency issues. See 
> /var/log/ambari-server/ambari-server-check-database.log for more details on 
> the consistency issues.
> ERROR: Exiting with exit code -1.
> REASON: Ambari Server java process has stopped. Please check the logs for 
> more information.
> {code}
> 
> DB log: ambari-server-check-database.log
> {code}
> 2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
> 2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
> configs that we got from db
> 2017-06-27 13:51:38,748  INFO - Getting services from metainfo
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KNOX
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KERBEROS
> 2017-06-27 13:51:38,749  INFO - Comparing required service configs from stack 
> with mapped service configs from db
> 2017-06-27 13:51:38,751 ERROR - Required config(s): product-info is(are) not 
> available for service SMARTSENSE with service config version 2 in cluster cl1
> 2017-06-27 13:51:38,751 ERROR - Required config(s): parquet-logging is(are) 
> not available for service HIVE with service config version 7 in cluster cl1
> 2017-06-27 13:51:38,751  INFO - Getting services which has mapped configs 
> which are not selected in clusterconfigmapping
> 2017-06-27 13:51:38,755  INFO - Checking Topology tables
> 2017-06-27 13:51:38,757  INFO - Checking for tables with large physical size
> 2017-06-27 13:51:38,759  INFO 

Re: Review Request 60998: ACTIVITY_ANALYZER Install failed: Error: Unable to run the custom hook script

[Eugene Chekanskiy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60998/#review181036
---


Ship it!




Ship It!

- Eugene Chekanskiy


On Липень 20, 2017, 1:04 після полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60998/
> ---
> 
> (Updated Липень 20, 2017, 1:04 після полудня)
> 
> 
> Review request for Ambari and Eugene Chekanskiy.
> 
> 
> Bugs: AMBARI-21535
> https://issues.apache.org/jira/browse/AMBARI-21535
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> STR:
> 
>   * Create ambari-qa and hbase user with UIDs less than 1000
>   * Navigate through UI install wizard. At customize servicespage Set the 
> checkbox for "Misc->Have Ambari manage UIDs" to true/checked so that after 
> deployment above created users will have UIDs >= 1000
>   * Go through install wizard to finish off deployment. But it fails at 
> Activity analyser install with the below error 
> 
> 
> {
>   "href" : 
> "http://172.27.25.210:8080/api/v1/clusters/cl1/requests/4/tasks/29";,
>   "Tasks" : {
> "attempt_cnt" : 1,
> "cluster_name" : "cl1",
> "command" : "INSTALL",
> "command_detail" : "ACTIVITY_ANALYZER INSTALL",
> "end_time" : 1500427251810,
> "error_log" : "/var/lib/ambari-agent/data/errors-29.txt",
> "exit_code" : 1,
> "host_name" : "ctr-e134-1499953498516-19756-01-05.hwx.site",
> "id" : 29,
> "output_log" : "/var/lib/ambari-agent/data/output-29.txt",
> "request_id" : 4,
> "role" : "ACTIVITY_ANALYZER",
> "stage_id" : 0,
> "start_time" : 1500427242346,
> "status" : "FAILED",
> "stderr" : "Traceback (most recent call last):\n  File 
> \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py\",
>  line 35, in \nBeforeAnyHook().execute()\n  File 
> \"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\",
>  line 329, in execute\nmethod(env)\n  File 
> \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py\",
>  line 29, in hook\nsetup_users()\n  File 
> \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py\",
>  line 60, in setup_users\nset_uid(params.smoke_user, 
> params.smoke_user_dirs)\n  File 
> \"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py\",
>  line 149, in set_uid\nnot_if = format(\"(test $(id -u {user}) -gt 1000) 
> || ({ignore_groupsusers_create_str})\"))\n  File 
> \"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 
> 155, in __init__\nself.env.run()\n  Fil
 e 
\"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", 
line 160, in run\nself.run_action(resource, action)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", 
line 124, in run_action\nprovider_action()\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\",
 line 262, in action_run\ntries=self.resource.tries, 
try_sleep=self.resource.try_sleep)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 
72, in inner\nresult = function(command, **kwargs)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 
102, in checked_call\ntries=tries, try_sleep=try_sleep, 
timeout_kill_strategy=timeout_kill_strategy)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 
150, in _call_wrapper\nresult = _call(command, **kwargs_copy)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/co
 re/shell.py\", line 303, in _call\nraise ExecutionFailed(err_msg, code, 
out, err)\nresource_management.core.exceptions.ExecutionFailed: Execution of 
'/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa 
/tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa
 0' returned 1. Failed to find Uid between 1000 and 2000\nError: Error: Unable 
to run the custom hook script ['/usr/bin/python', 
'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py',
 'ANY', '/var/lib/ambari-agent/data/command-29.json', 
'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', 
'/var/lib/ambari-agent/data/structured-out-29.json', 'INFO', 
'/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']",
> "stdout" : "2017-07-19 01:20:49,237 - Stack Feature Version Info: 
> Cluster Stack=2.6, Cluster Current Version=None, Command Stack=None, Command 
> Version=None-> 2.6\n2017-07-19 01:20:49,286 - Using hadoo

Review Request 60998: ACTIVITY_ANALYZER Install failed: Error: Unable to run the custom hook script

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60998/
---

Review request for Ambari and Eugene Chekanskiy.


Bugs: AMBARI-21535
https://issues.apache.org/jira/browse/AMBARI-21535


Repository: ambari


Description
---

STR:

  * Create ambari-qa and hbase user with UIDs less than 1000
  * Navigate through UI install wizard. At customize servicespage Set the 
checkbox for "Misc->Have Ambari manage UIDs" to true/checked so that after 
deployment above created users will have UIDs >= 1000
  * Go through install wizard to finish off deployment. But it fails at 
Activity analyser install with the below error 


{
  "href" : 
"http://172.27.25.210:8080/api/v1/clusters/cl1/requests/4/tasks/29";,
  "Tasks" : {
"attempt_cnt" : 1,
"cluster_name" : "cl1",
"command" : "INSTALL",
"command_detail" : "ACTIVITY_ANALYZER INSTALL",
"end_time" : 1500427251810,
"error_log" : "/var/lib/ambari-agent/data/errors-29.txt",
"exit_code" : 1,
"host_name" : "ctr-e134-1499953498516-19756-01-05.hwx.site",
"id" : 29,
"output_log" : "/var/lib/ambari-agent/data/output-29.txt",
"request_id" : 4,
"role" : "ACTIVITY_ANALYZER",
"stage_id" : 0,
"start_time" : 1500427242346,
"status" : "FAILED",
"stderr" : "Traceback (most recent call last):\n  File 
\"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py\",
 line 35, in \nBeforeAnyHook().execute()\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py\",
 line 329, in execute\nmethod(env)\n  File 
\"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py\",
 line 29, in hook\nsetup_users()\n  File 
\"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py\",
 line 60, in setup_users\nset_uid(params.smoke_user, 
params.smoke_user_dirs)\n  File 
\"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py\",
 line 149, in set_uid\nnot_if = format(\"(test $(id -u {user}) -gt 1000) || 
({ignore_groupsusers_create_str})\"))\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/base.py\", line 
155, in __init__\nself.env.run()\n  File 
 \"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", 
line 160, in run\nself.run_action(resource, action)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/environment.py\", 
line 124, in run_action\nprovider_action()\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py\",
 line 262, in action_run\ntries=self.resource.tries, 
try_sleep=self.resource.try_sleep)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 
72, in inner\nresult = function(command, **kwargs)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 
102, in checked_call\ntries=tries, try_sleep=try_sleep, 
timeout_kill_strategy=timeout_kill_strategy)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core/shell.py\", line 
150, in _call_wrapper\nresult = _call(command, **kwargs_copy)\n  File 
\"/usr/lib/python2.6/site-packages/resource_management/core
 /shell.py\", line 303, in _call\nraise ExecutionFailed(err_msg, code, out, 
err)\nresource_management.core.exceptions.ExecutionFailed: Execution of 
'/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa 
/tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa
 0' returned 1. Failed to find Uid between 1000 and 2000\nError: Error: Unable 
to run the custom hook script ['/usr/bin/python', 
'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py',
 'ANY', '/var/lib/ambari-agent/data/command-29.json', 
'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', 
'/var/lib/ambari-agent/data/structured-out-29.json', 'INFO', 
'/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']",
"stdout" : "2017-07-19 01:20:49,237 - Stack Feature Version Info: 
Cluster Stack=2.6, Cluster Current Version=None, Command Stack=None, Command 
Version=None-> 2.6\n2017-07-19 01:20:49,286 - Using hadoop conf dir: 
/usr/hdp/current/hadoop-client/conf\nUser Group mapping (user_group) is missing 
in the hostLevelParams\n2017-07-19 01:20:49,288 - Group['hadoop'] 
{}\n2017-07-19 01:20:49,292 - Group['users'] {}\n2017-07-19 01:20:49,293 - 
File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': 
StaticFile('changeToSecureUid.sh'), 'mode': 0555}\n2017-07-19 01:20:49,297 - 
Writing File['/var/lib/ambari-agent/tmp/changeUid.sh'] because it doesn't 
exist\n2017-07-19 01:20:49,297 - Changing permission for 
/var/lib/ambari-age

Re: Review Request 60986: AMBARI-21528. Zookeeper server has incorrect memory setting, missing m in Xmx value

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60986/#review181035
---


Ship it!




Ship It!

- Tim Thorpe


On July 20, 2017, 12:13 a.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60986/
> ---
> 
> (Updated July 20, 2017, 12:13 a.m.)
> 
> 
> Review request for Ambari, Di Li, Jonathan Hurley, Sumit Mohanty, Sid Wagle, 
> and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21528
> https://issues.apache.org/jira/browse/AMBARI-21528
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Repro Steps:
> 
> * Installed BI 4.2.0 cluster on IBM Ambari 2.2.2 with Zookeeper
> * Upgraded Ambari to 2.5.2.0-146
> * Registered HDP 2.6.2.0 repo, installed packages
> * Ran service checks
> * Started Express Upgrade
> 
> Result: _Service Check ZooKeeper_ step failed with {{KeeperErrorCode = 
> ConnectionLoss for /zk_smoketest}}
> 
> This was caused by Zookeeper dying immediately during restart:
> ```
> Error occurred during initialization of VM
> Too small initial heap
> ```
> 
> Before EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/iop/current/zookeeper-server
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024m
> export JAVA=$JAVA_HOME/bin/java
> export CLASSPATH=$CLASSPATH:/usr/share/zookeeper/*
> ```
> 
> After EU
> ```
> export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64
> export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-client
> export ZOO_LOG_DIR=/var/log/zookeeper
> export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
> export SERVER_JVMFLAGS=-Xmx1024
> export JAVA=$JAVA_HOME/bin/java
> ```
> 
> Note missing "m" in memory setting.
> 
> zookeeper-env template contains,
> ```
> export SERVER_JVMFLAGS={{zk_server_heapsize}}
> ```
> 
> In this cluster, zookeeper-env contains,
> zk_server_heapsize: "1024"
> 
> While the params_linux.py file has some inconsistencies with appending the 
> letter "m".
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024m"))
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> Instead, it should be,
> ```
> zk_server_heapsize_value = 
> str(default('configurations/zookeeper-env/zk_server_heapsize', "1024"))
> zk_server_heapsize_value = zk_server_heapsize_value.strip()
> if len(zk_server_heapsize_value) > 0 and not 
> zk_server_heapsize_value[-1].isdigit():
>   zk_server_heapsize_value = zk_server_heapsize_value + "m"
> zk_server_heapsize = format("-Xmx{zk_server_heapsize_value}")
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py
>  0780d2e 
> 
> 
> Diff: https://reviews.apache.org/r/60986/diff/2/
> 
> 
> Testing
> ---
> 
> Python unit tests passed,
> 
> --
> Total run:1161
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Review Request 60995: Namenode restart - PID file delete happens before the call to check status

[Dmitro Lisnichenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60995/
---

Review request for Ambari, Jonathan Hurley, Nate Cole, and Sumit Mohanty.


Bugs: AMBARI-21532
https://issues.apache.org/jira/browse/AMBARI-21532


Repository: ambari


Description
---

PID file delete happens before the call to check status.

{code}
...
2017-07-06 00:03:21,004 - File['/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid'] 
{'action': ['delete']}
2017-07-06 00:05:21,103 - Waiting for actual component stop
2017-07-06 00:05:21,104 - Pid file 
/var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid is empty or does not exist
{code}


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/check_process_status.py
 7961f008ca 
  ambari-common/src/main/python/resource_management/libraries/script/script.py 
2b374c5cff 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py
 d861ba93a5 
  
ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/utils.py
 53774c661a 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/package/scripts/utils.py
 7dcbca8d5b 


Diff: https://reviews.apache.org/r/60995/diff/1/


Testing
---

mvn clean test


Thanks,

Dmitro Lisnichenko

Re: Review Request 60994: Client component restart fails after Ambari upgrade while running custom hook script on Suse 11

[Vitalyi Brodetskyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60994/#review181033
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Липень 20, 2017, 10:20 до полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60994/
> ---
> 
> (Updated Липень 20, 2017, 10:20 до полудня)
> 
> 
> Review request for Ambari and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21531
> https://issues.apache.org/jira/browse/AMBARI-21531
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Seen in two cluster with Suse 11 SP4 OS
> 
> **STR**
> 
>   1. Deployed cluster with Ambari version: 2.4.2.0-136 and HDP version: 
> 2.5.3.0-37 (secure cluster, wire encryption enabled one cluster, disabled on 
> second cluster)
>   2. Upgrade Ambari to 2.5.2.0-147 (hash: 
> be3a875972224d7eb420c783a9f2cbdc7157)
>   3. Regenerate keytabs post upgrade and then try to restart all services
> 
> **Result:**  
> Observed errors at start of Falcon, HBase, Atlas clients:
> 
> 
> 
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py",
>  line 35, in 
> BeforeAnyHook().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 329, in execute
> method(env)
>   File 
> "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py",
>  line 29, in hook
> setup_users()
>   File 
> "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py",
>  line 51, in setup_users
> groups = params.user_to_groups_dict[user],
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 
> 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/accounts.py",
>  line 82, in action_create
> shell.checked_call(command, sudo=True)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 
> 72, in inner
> result = function(command, **kwargs)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 
> 102, in checked_call
> tries=tries, try_sleep=try_sleep, 
> timeout_kill_strategy=timeout_kill_strategy)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 
> 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 
> 303, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 
> 'usermod -u 1002 -G hadoop,hadoop -g hadoop hive' returned 11. usermod: 
> `hadoop' is primary group name.
> usermod: `hadoop' is primary group name.
> usermod: UID 1002 is not unique.
> Error: Error: Unable to run the custom hook script ['/usr/bin/python', 
> '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py',
>  'ANY', '/var/lib/ambari-agent/data/command-864.json', 
> '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', 
> '/var/lib/ambari-agent/data/structured-out-864.json', 'INFO', 
> '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']Error: Error: Unable to run 
> the custom hook script ['/usr/bin/python', 
> '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py',
>  'START', '/var/lib/ambari-agent/data/command-864.json', 
> '/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', 
> '/var/lib/ambari-agent/data/structured-out-864.json', 'INFO', 
> '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']
> 
> 
> Suspect something to do with TLS v1 protocol on Suse 11.4
> 
> Cluster:  (alive for 48h)
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/test/python/resource_management/TestUserResource.py 
> 97d992e 
>   ambari-common/src/main/python/resource_management/core/base.py 1500e1f 
>   
> ambari-common/src/main/python/resource_management/core/providers/accounts.py 
> c4f2496 
>   
> ambari-common/src/main/python/resource_management/core/resources/accounts.py 
> 4ee2c57 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py
>  886bc45 
>

Review Request 60994: Client component restart fails after Ambari upgrade while running custom hook script on Suse 11

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60994/
---

Review request for Ambari and Vitalyi Brodetskyi.


Bugs: AMBARI-21531
https://issues.apache.org/jira/browse/AMBARI-21531


Repository: ambari


Description
---

Seen in two cluster with Suse 11 SP4 OS

**STR**

  1. Deployed cluster with Ambari version: 2.4.2.0-136 and HDP version: 
2.5.3.0-37 (secure cluster, wire encryption enabled one cluster, disabled on 
second cluster)
  2. Upgrade Ambari to 2.5.2.0-147 (hash: 
be3a875972224d7eb420c783a9f2cbdc7157)
  3. Regenerate keytabs post upgrade and then try to restart all services

**Result:**  
Observed errors at start of Falcon, HBase, Atlas clients:




Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py",
 line 35, in 
BeforeAnyHook().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 329, in execute
method(env)
  File 
"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py",
 line 29, in hook
setup_users()
  File 
"/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py",
 line 51, in setup_users
groups = params.user_to_groups_dict[user],
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", 
line 155, in __init__
self.env.run()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 160, in run
self.run_action(resource, action)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", 
line 124, in run_action
provider_action()
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/providers/accounts.py",
 line 82, in action_create
shell.checked_call(command, sudo=True)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, 
in inner
result = function(command, **kwargs)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, 
in checked_call
tries=tries, try_sleep=try_sleep, 
timeout_kill_strategy=timeout_kill_strategy)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, 
in _call_wrapper
result = _call(command, **kwargs_copy)
  File 
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, 
in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'usermod 
-u 1002 -G hadoop,hadoop -g hadoop hive' returned 11. usermod: `hadoop' is 
primary group name.
usermod: `hadoop' is primary group name.
usermod: UID 1002 is not unique.
Error: Error: Unable to run the custom hook script ['/usr/bin/python', 
'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY/scripts/hook.py',
 'ANY', '/var/lib/ambari-agent/data/command-864.json', 
'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY', 
'/var/lib/ambari-agent/data/structured-out-864.json', 'INFO', 
'/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']Error: Error: Unable to run 
the custom hook script ['/usr/bin/python', 
'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START/scripts/hook.py',
 'START', '/var/lib/ambari-agent/data/command-864.json', 
'/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-START', 
'/var/lib/ambari-agent/data/structured-out-864.json', 'INFO', 
'/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1', '']


Suspect something to do with TLS v1 protocol on Suse 11.4

Cluster:  (alive for 48h)


Diffs
-

  ambari-agent/src/test/python/resource_management/TestUserResource.py 97d992e 
  ambari-common/src/main/python/resource_management/core/base.py 1500e1f 
  ambari-common/src/main/python/resource_management/core/providers/accounts.py 
c4f2496 
  ambari-common/src/main/python/resource_management/core/resources/accounts.py 
4ee2c57 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py
 886bc45 
  
ambari-server/src/test/python/stacks/2.0.6/hooks/before-ANY/test_before_any.py 
1d2351f 


Diff: https://reviews.apache.org/r/60994/diff/1/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 60943: Log Search Solr output properties should be provided by the Config API

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60943/
---

(Updated July 20, 2017, 9:54 a.m.)


Review request for Ambari, Oliver Szabo and Robert Nettleton.


Changes
---

Get number of shards and shard names from solr state json from zookeeper
remove codes for creating solr client using urls as it wasn't supported anyway 
by the server


Bugs: AMBARI-21507
https://issues.apache.org/jira/browse/AMBARI-21507


Repository: ambari


Description
---

The Solr output properties should be persisted into the Config API by the Log 
Search Server as the output is initialized, and fetched by he Log Feeders from 
there


Diffs (updated)
-

  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfig.java
 6c5cefd 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/OutputConfigMonitor.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/outputconfig/OutputProperties.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/model/outputconfig/OutputSolrProperties.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass1.java
 28844d5 
  
ambari-logsearch/ambari-logsearch-config-api/src/test/java/org/apache/ambari/logsearch/config/api/LogSearchConfigClass2.java
 5934fa6 
  
ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogSearchConfigZK.java
 fdd8ed6 
  
ambari-logsearch/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/model/outputconfig/impl/OutputSolrPropertiesImpl.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
 59c2a22 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
 5bf074c 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
 5356159 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
 09fc3f5 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java
 f1002ae 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/Output.java
 65b9e19 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManager.java
 4d6c43b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputSolr.java
 162a7f8 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputSolrTest.java
 8985110 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrAuditLogPropsConfig.java
 c569a27 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrEventHistoryPropsConfig.java
 975e6a7 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrPropsConfig.java
 ceddf7e 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrServiceLogPropsConfig.java
 e5039d5 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/LogSearchConfigConfigurer.java
 c34dce6 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/configurer/SolrCollectionConfigurer.java
 f2d022e 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/AuditSolrDao.java
 3eea08f 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/ServiceLogsSolrDao.java
 308ef1f 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/SolrDaoBase.java
 b30b6ef 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/handler/CreateCollectionHandler.java
 752a1e1 
  
ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/ShipperConfigManager.java
 2c143c0 
  ambari-logsearch/docker/test-config/logfeeder/shipper-conf/output.config.json 
55fd36c 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
 b4502d6 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/properties/output.config.json.j2
 214e5ba 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/service_advisor.py
 fc49824 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/Up

Re: Review Request 60972: Include/exclude files update is triggered when master is not started if cluster is deployed via BP

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60972/#review181021
---


Ship it!




Ship It!

- Andrew Onischuk


On July 19, 2017, 2:16 p.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60972/
> ---
> 
> (Updated July 19, 2017, 2:16 p.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Sumit Mohanty, Sid Wagle, and 
> Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21521
> https://issues.apache.org/jira/browse/AMBARI-21521
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Include/exclude files update is triggered when master component is not 
> started if cluster is deployed via BP.
> BP deployments do not guarantee components start order.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  38842fa 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerImplTest.java
>  1f2c332 
> 
> 
> Diff: https://reviews.apache.org/r/60972/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>