date:20160319


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44986/#review124222
---


Ship it!




Ship It!

- Nate Cole


On March 17, 2016, 7 p.m., Nahappan Somasundaram wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44986/
> ---
> 
> (Updated March 17, 2016, 7 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15474
> https://issues.apache.org/jira/browse/AMBARI-15474
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-15474: Listen for changes to auto-start configuration and send them to 
> the agent during heartbeats.
> 
> In HeartbeatHandler::handleHeartBeat(), send the recovery configuration to 
> the agent. Instead of pulling the recovery configuration from the application 
> cache, implement change events and get notified whenever there are changes to 
> the recovery configuration.
> 
> **Changes**:
> 1. Instead of sending the recovery configuration to the agent during every 
> heartbeat (10 seconds interval), send the configuration only when changes are 
> detected.
> 2. Used AmbariEventPublisher to publish the changes. RecoveryConfigHelper is 
> the subscriber which listens to changes to the configuration.
> 3. RecoveryConfigHelper maintains a map of Cluster-->Host-->Timestamp which 
> is the last updated timestamp of the recovery configuration. This timestamp 
> is used as a token and is sent to the agent with the configuration. The agent 
> provides this timestamp during each heartbeat. If the timestamp from the 
> heartbeat is the same as the one on the server side, no configuration is sent 
> to the agent.
> 4. Whenever changes are detected, the timestamp is invalidated. During the 
> next heartbeat, the recovery configuration is read from the DB and sent to 
> the agent along with the latest timestamp.
> 5. Added new unit tests to test each configuration change event.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py 
> c1c16ac97cca0d2677da2e26eee7b4949a4bf15c 
>   ambari-agent/src/main/python/ambari_agent/Heartbeat.py 
> b28644c834a2387d2fb0ad17d104224e830a0245 
>   ambari-agent/src/main/python/ambari_agent/RecoveryManager.py 
> ed537ca6928ce376c5f3e906f7a47c3f1919e309 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeat.java 
> 1430aa20cf68db071be85c4ad2ebb9acdaccecc0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  3a80803724bcb6ca9e4ec875aa17c9ac1c66fbe8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
>  617b04b5b8768ba28a3c8ecb6e5e00601f153396 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfig.java
>  c2c18462c586292caaf40bdb6a25a8fe9a39d76c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
>  92a622117c71ddfc3bd2562c04ee525491bd6ffd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/AmbariEvent.java 
> 78731854fe80c3d6087a73ebf00b0ffe9e287354 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ClusterConfigChangedEvent.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentInstalledEvent.java
>  2ce01236b3bb1ab4934fe456e5669fa474987426 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentRecoveryChangeEvent.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentUninstalledEvent.java
>  19712cdb18592816f82898cd7aa08a3600c7e1f4 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ConfigImpl.java 
> 2cc3d0013140c9c7d3453d7c5e05c3aa02bc5794 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentImpl.java
>  4be1c21bf41416681be836716ce38305cc3f287e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
>  3a0075e3a3c90088ca6bd2d9eca48ef50db9fdb3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  b1f94e38f97550123dd366984438b0bbf9c2826c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/RecoveryConfigHelperTest.java
>  c86b95a7cd10b0720f0a3b9676f65a8bd68dd6c9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/events/listeners/upgrade/HostVersionOutOfSyncListenerTest.java
>  1bf9d8336533bc677384314f998f10aa2f10504e 
> 
> Diff: https://reviews.apache.org/r/44986/diff/
> 
> 
> Testing
> ---
> 
> **1. mvn clean install **
> 
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]

Review Request 45041: Strange paths in ambari-server setup

2016-03-19 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45041/
---

Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Dmytro Sen.


Bugs: AMBARI-15483
https://issues.apache.org/jira/browse/AMBARI-15483


Repository: ambari


Description
---

ambari-server setup command shows strange paths with double slashes in front:
Successfully installed JDK to //usr/jdk64/
Downloading JCE Policy archive from 
http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip to 
//var/lib/ambari-server/resources/jce_policy-8.zip
Successfully downloaded JCE Policy archive to 
//var/lib/ambari-server/resources/jce_policy-8.zip
Installing JCE policy...
Completing setup...
Configuring database...
Enter advanced database configuration y/n ?
Configuring database...
Default properties detected. Using built-in database.
WARNING: Command chown None //etc/ambari-server/conf/password.dat returned exit 
code //etc/ambari-server/conf/password.dat with message: chown: invalid user: 
`None'


Diffs
-

  ambari-server/src/main/python/ambari_server/serverConfiguration.py 29187bd 

Diff: https://reviews.apache.org/r/45041/diff/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 44916: StackAdvisor needs to set config map for ams-grafana-env to meet non-null expectations


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44916/
---

(Updated March 17, 2016, 2:56 p.m.)


Review request for Ambari, Dmytro Sen, Jonathan Hurley, Nate Cole, and Sid 
Wagle.


Bugs: AMBARI-15442
https://issues.apache.org/jira/browse/AMBARI-15442


Repository: ambari


Description
---

The StackAdvisor needs to set a config map for the `ams-grafana-env`, even it 
is empty, to meet non-null expectations in Kerberos and Blueprint logic. 

To to this, the following line needs to be added to the StackAdvisor code:

#stacks/HDP/2.0.6/services/stack_advisor.py:473
```
putGrafanaProperty = self.putProperty(configurations, "ams-grafana-env", 
services)
```


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/recommendations/RecommendationResponse.java
 4a66677 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
54d6d6e 
  ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 59eff4d 

Diff: https://reviews.apache.org/r/44916/diff/


Testing (updated)
---

Manually tested in cluster Kerbrerized using the Ambari UI.

# Local test results:
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 1:04:54.682s
[INFO] Finished at: Thu Mar 17 14:54:27 EDT 2016
[INFO] Final Memory: 74M/1855M
[INFO] 

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 44565: Hive view: Visual Explain does not work when there are multiple statements in the query editor

2016-03-19 Thread Pallav Kulshreshtha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44565/
---

(Updated March 17, 2016, 4:07 p.m.)


Review request for Ambari, DIPAYAN BHOWMICK, Srimanth Gunturi, and Yusaku Sako.


Bugs: AMBARI-15346
https://issues.apache.org/jira/browse/AMBARI-15346


Repository: ambari


Description
---

Adding the "Explain | Explain Formatted" only for the last query. Make some 
modification in "updateValue" function of query editor component to handle this 
change.


Diffs (updated)
-

  
contrib/views/hive/src/main/resources/ui/hive-web/app/components/query-editor.js
 10c6bb3 
  contrib/views/hive/src/main/resources/ui/hive-web/app/controllers/index.js 
a056759 

Diff: https://reviews.apache.org/r/44565/diff/


Testing
---

manual testing done. No extra unit test case needed.


Thanks,

Pallav Kulshreshtha

Re: Review Request 44868: Changed HAWQ directory descriptions


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44868/#review123918
---


Ship it!




Ship It!

- Jayush Luniya


On March 15, 2016, 10:51 p.m., Alexander Denissov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44868/
> ---
> 
> (Updated March 15, 2016, 10:51 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, bhuvnesh chaudhary, Goutam 
> Tadi, jun aoki, Jayush Luniya, Lav Jain, Oleksandr Diachenko, and Sumit 
> Mohanty.
> 
> 
> Bugs: AMBARI-15433
> https://issues.apache.org/jira/browse/AMBARI-15433
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Changed HAWQ directory descriptions
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  d3aa0ed 
> 
> Diff: https://reviews.apache.org/r/44868/diff/
> 
> 
> Testing
> ---
> 
> manual.
> 
> 
> Thanks,
> 
> Alexander Denissov
> 
>

Re: Review Request 44987: Editing tmp dir from single to multiple dir doesn't trigger the backend change


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44987/#review124309
---


Ship it!




Ship It!

- bhuvnesh chaudhary


On March 18, 2016, 10:17 p.m., jun aoki wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44987/
> ---
> 
> (Updated March 18, 2016, 10:17 p.m.)
> 
> 
> Review request for Ambari.
> 
> 
> Bugs: AMBARI-15475
> https://issues.apache.org/jira/browse/AMBARI-15475
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Modified "HAWQ Segment Temp Directory” from one [/d1/tmp/hawqsegment] to many 
> [/d1/tmp/hawqsegment,/d2/tmp/hawqsegment,/d3/tmp/hawqsegment,/d4/tmp/hawqsegment,/d5/tmp/hawqsegment]
> and the change is not propagated to segments and the folders are not created.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/common.py
>  d62332d 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/hawqsegment.py
>  0a597b6 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/master_helper.py
>  9482b35 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/params.py
>  1bdc5aa 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/utils.py
>  dc6c203 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqmaster.py 14b5a5f 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqsegment.py 6ca83b7 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqstandby.py eaba496 
> 
> Diff: https://reviews.apache.org/r/44987/diff/
> 
> 
> Testing
> ---
> 
> manually tested.
> 
> test_configure_default (test_hawqsegment.TestHawqSegment) ... ok
> test_install_default (test_hawqsegment.TestHawqSegment) ... ok
> test_start_default (test_hawqsegment.TestHawqSegment) ... ok
> test_stop_default (test_hawqsegment.TestHawqSegment) ... ok
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_configure_default (test_hawqstandby.TestHawqStandby) ... ok
> test_install_default (test_hawqstandby.TestHawqStandby) ... ok
> test_start_default (test_hawqstandby.TestHawqStandby) ... 2016-03-17 
> 15:33:42,969 - Standby host c6401.ambari.apache.org initialized
> ok
> test_stop_default (test_hawqstandby.TestHawqStandby) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_start_default (test_hawqmaster.TestHawqMaster) ... 2016-03-17 
> 15:33:43,025 - Active master c6401.ambari.apache.org initialized
> ok
> test_start_localmaster (test_hawqmaster.TestHawqMaster) ... 2016-03-17 
> 15:33:43,049 - Master c6401.ambari.apache.org started
> ok
> test_stop_default (test_hawqmaster.TestHawqMaster) ... ok
> 
> 
> Thanks,
> 
> jun aoki
> 
>

Re: Review Request 44958: Increase Ambari Server Perm gen default value


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44958/#review124042
---


Fix it, then Ship it!





ambari-server/src/main/python/ambari_server_main.py (line 89)


I am not familiar with these options, but it apears like no one really 
knows what they do.  See https://dzone.com/articles/busting-permgen-myths for 
an example. 

Do we really think that this is necessary?  Do we know how this will affect 
performance?


- Robert Levas


On March 17, 2016, 10:57 a.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44958/
> ---
> 
> (Updated March 17, 2016, 10:57 a.m.)
> 
> 
> Review request for Ambari, Laszlo Puskas, Oliver Szabo, Robert Levas, Sandor 
> Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-15465
> https://issues.apache.org/jira/browse/AMBARI-15465
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Increased permGen size to 128m, also added **-XX:+CMSPermGenSweepingEnabled** 
> and **-XX:+CMSClassUnloadingEnabled** to allow jvm to keep permGen space 
> clean.
> 
> 
> Diffs
> -
> 
>   ambari-server/conf/unix/ambari-env.sh e9fdcdd 
>   ambari-server/src/main/python/ambari_server_main.py 00e084b 
> 
> Diff: https://reviews.apache.org/r/44958/diff/
> 
> 
> Testing
> ---
> 
> Manual testing:
> [root@c6401 sbin]# ps -ef | grep ambari-server
> root 20769 1 27 14:47 pts/000:00:24 
> //usr/jdk64/jdk1.8.0_60/bin/java -server -XX:NewRatio=3 
> -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit 
> -XX:CMSInitiatingOccupancyFraction=60 **-XX:+CMSPermGenSweepingEnabled 
> -XX:+CMSClassUnloadingEnabled** -Dsun.zip.disableMemoryMapping=true -Xms512m 
> -Xmx2048m **-XX:MaxPermSize=128m** 
> -Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
> -Djava.security.krb5.conf=/etc/krb5.conf 
> -Djavax.security.auth.useSubjectCredsOnly=false -Xms512m -Xmx2048m 
> -Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
> -Djava.security.krb5.conf=/etc/krb5.conf 
> -Djavax.security.auth.useSubjectCredsOnly=false -cp 
> //etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar
>  org.apache.ambari.server.controller.AmbariServer
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Review Request 44931: Use Version Definition value for package-version when installing


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44931/
---

Review request for Ambari, Alejandro Fernandez and Jonathan Hurley.


Bugs: AMBARI-15447
https://issues.apache.org/jira/browse/AMBARI-15447


Repository: ambari


Description
---

The version defintion file defines a package version that should be used when 
installing packages.  The current calculation cannot be considered reliable, 
but should stay as a contingency for when NOT using a version definition.


Diffs
-

  ambari-common/src/main/python/resource_management/libraries/script/script.py 
a8098a0 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 402a338 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 e32e2f9 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
 07e62b3 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostStackVersionResourceProvider.java
 1cd9c0a 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
 3398709 
  
ambari-server/src/main/java/org/apache/ambari/server/state/repository/Release.java
 6bcedf5 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProviderTest.java
 dea83a1 
  ambari-server/src/test/python/custom_actions/TestInstallPackages.py f022c80 
  ambari-server/src/test/resources/hbase_version_test.xml 9df07ed 

Diff: https://reviews.apache.org/r/44931/diff/


Testing
---

Manual.  Automated pending.


Thanks,

Nate Cole

Review Request 44916: StackAdvisor needs to set config map for ams-grafana-env to meet non-null expectations


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44916/
---

Review request for Ambari, Dmytro Sen, Jonathan Hurley, Nate Cole, and Sid 
Wagle.


Bugs: AMBARI-15442
https://issues.apache.org/jira/browse/AMBARI-15442


Repository: ambari


Description
---

The StackAdvisor needs to set a config map for the `ams-grafana-env`, even it 
is empty, to meet non-null expectations in Kerberos and Blueprint logic. 

To to this, the following line needs to be added to the StackAdvisor code:

#stacks/HDP/2.0.6/services/stack_advisor.py:473
```
putGrafanaProperty = self.putProperty(configurations, "ams-grafana-env", 
services)
```


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
8c18421 

Diff: https://reviews.apache.org/r/44916/diff/


Testing
---

Manually tested in cluster Kerbrerized using the Ambari UI.


Thanks,

Robert Levas

Re: Review Request 45035: Restarting HDFS Before Upgrade Finalizing Does Not Supply the rollingUpgrade Flag


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45035/#review124232
---




ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
 (line 441)


I think adding a new column is overkill. If EU is suspended while stopping 
services, then HDFS has not yet been prepared or restarted with the 
-rollingUpgrade option.

I wonder if there's a way to detect on the python side that NameNode is in 
the middle of an upgrade and hence should be restarted with that flag.
Could even use a marker file that gets deleted after Finalize.


- Alejandro Fernandez


On March 18, 2016, 3:20 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45035/
> ---
> 
> (Updated March 18, 2016, 3:20 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.
> 
> 
> Bugs: AMBARI-15482
> https://issues.apache.org/jira/browse/AMBARI-15482
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During an upgrade, the HDFS NameNode(s) are restarted with the 
> {{-rollingUpgrade}} flag. However, it's possible to get to the end of an 
> upgrade and decide to "Finalize Later".
> 
> This allows the cluster to run in the upgraded state before committing to the 
> upgrade. Full cluster control is returned via the Ambari web client.
> 
> Administrators can then decide to restart a NameNode. Upon restarting the 
> NameNode, it will produce an error that it was not started with the 
> {{rollingUpgrade}} flag. 
> 
> It seems that as long as an upgrade has not been finalized, the NameNode(s) 
> must be started with the {{rollingUpgrade}} to allow them to function 
> properly. 
> 
> STR:
> - Perform a rolling upgrade from HDP 2.2 to 2.3 (or 2.3 to 2.4); as long as 
> there is a major version change.
> - Before finalization, say "Finalize Later". All services should be up and 
> green.
> - Now restart a NameNode
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  9ea541e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
>  4ef215c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  303f3a4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  07061e1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
>  3f1a52b 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> ed3c772 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  1c7ff61 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
>  84bb9f3 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql a85202d 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 9b4810c 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql cc3d197 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 07c786d 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> ab6dc93 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 8e91fde 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 440ca44 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
>  e4c8c9c 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  02905ec 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
>  905802f 
>   ambari-web/app/controllers/main/admin/stack_and_upgrade_controller.js 
> 2dceccc 
>   ambari-web/app/utils/ajax/ajax.js 29d0715 
> 
> Diff: https://reviews.apache.org/r/45035/diff/
> 
> 
> Testing
> ---
> 
> Pending...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Review Request 44972: Improve error logging for install errors during blueprint deployments.

2016-03-19 Thread Amruta Borkar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44972/
---

Review request for Ambari, Di Li and Sumit Mohanty.


Bugs: AMBARI-15412
https://issues.apache.org/jira/browse/AMBARI-15412


Repository: ambari


Description
---

Improve error logging for install errors during blueprint deployments. 

Currently a severe error during install of a service component gets logged as a 
WARNing.

E.g.:

09 Mar 2016 12:11:45,881 WARN [qtp-ambari-agent-146] HeartBeatHandler:603 - 
Operation failed - may be retried. Service component host: KAFKA_BROKER, host: 
hdtest159.svl.ibm.com Action id12-0


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 24fea22 

Diff: https://reviews.apache.org/r/44972/diff/


Testing
---

There are no JUnit test cases, but attached the screenshot which shows modified 
output.


File Attachments


Patch
  
https://reviews.apache.org/media/uploaded/files/2016/03/17/2110d517-0cf4-4477-87e7-935c963e04de__AMBARI-15412.patch
Output
  
https://reviews.apache.org/media/uploaded/files/2016/03/17/923d218f-7ca4-4439-b42f-743511936f94__AMBARI-15412_output.png


Thanks,

Amruta Borkar

Review Request 45030: Replace 0.0.0.0 with hostname in ams-site/timeline.metrics.service.webapp.address

2016-03-19 Thread Dmytro Sen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45030/
---

Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sid Wagle.


Bugs: AMBARI-15480
https://issues.apache.org/jira/browse/AMBARI-15480


Repository: ambari


Description
---

We need collector hostname in configs. 
Replace 0.0.0.0 with hostname in 
ams-site/timeline.metrics.service.webapp.address
Implement changes for blueprints, stack_advisor, upgrade catalog


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
 e73e38f 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
 e10585e 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/params.py
 3b8505b 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
 45c1811 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
 12c0f25 
  
ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/params.py
 5ec879c 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
 b517eeb 
  
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/params.py
 f75d578 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py
 a022282 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
 7a023cf 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
ee2007f 
  
ambari-server/src/main/resources/stacks/HDPWIN/2.1/hooks/before-START/scripts/params.py
 469e709 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
 ef5dc76 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java
 859513b 
  ambari-server/src/test/python/stacks/2.0.6/configs/default.json fe92b5b 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_ams_embedded.json 
688dbe0 
  ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 
4648ffc 
  ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 9b2da28 
  
ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json
 68afb68 
  ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 
79c5580 
  ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 
a7057db 
  ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json 
390f9ce 
  ambari-server/src/test/python/stacks/2.3/configs/ats_1_5.json 3fe1235 

Diff: https://reviews.apache.org/r/45030/diff/


Testing
---

Unit tests passed


Thanks,

Dmytro Sen

Re: Review Request 44687: Update RCO : PXF should start after HDFS

2016-03-19 Thread Tim Thorpe



> On March 11, 2016, 9:06 a.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json, 
> > line 14
> > 
> >
> > RCO can be defined at service level 
> > (https://issues.apache.org/jira/browse/AMBARI-9363). Instead of updating 
> > RCO of HDP stack, can you move the HAWQ and PXF specific ordering into the 
> > service?
> 
> bhuvnesh chaudhary wrote:
> Thanks Jayush for the suggestion, let me try that.
> However, RCO at service level will be applicable only for trunk, for 2.2 
> branch we still need to update HDP rco. correct ? 
> If so i will remove trunk tag from the list of branches, and keep this 
> pathc for 2.2.0 only.
> 
> bhuvnesh chaudhary wrote:
> Tried putting role_command_order.json under 
> /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json.
>  However, it appears that its not loaded due to which HAWQMASTER was started 
> before NAMENODE and it failed. Same for PXF.
> Here are the contents of RCO: 
> [root@c6401 ~]# cat 
> /var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/role_command_order.json
> {
>   "_comment" : "Record format:",
>   "_comment" : "blockedRole-blockedCommand: 
> [blockerRole1-blockerCommand1, blockerRole2-blockerCommand2, ...]",
>   "general_deps" : {
> "_comment" : "dependencies for all cases",
> "HAWQMASTER-START" : ["NAMENODE-START", "DATANODE-START", 
> "NODEMANAGER-START"],
> "HAWQSTANDBY-START" : ["HAWQMASTER-START"],
> "HAWQSEGMENT-START" : ["HAWQMASTER-START", "HAWQSTANDBY-START"],
> "HAWQ_SERVICE_CHECK-SERVICE_CHECK" : ["HAWQSEGMENT-START", 
> "HDFS_SERVICE_CHECK-SERVICE_CHECK", "YARN_SERVICE_CHECK-SERVICE_CHECK", 
> "PXF_SERVICE_CHECK-SERVICE_CHECK"]
>   }
> }
> 
> Checked ambari-server.log, it does not show that this RCO is loaded but 
> shows the others under /var/lib/ambari-server/resources/stacks which are 
> loaded, will need to debug further for the root cause. 
> Adding Tim Thorpe as well to the reviewers list.
> 
> Jayush Luniya wrote:
> @Bhuvnesh
> Go ahead with your changes without the refactoring in trunk and 2.2 
> branch. I dont want to block you on it. Lets handle the refactoring in a 
> separate JIRA. @Tim Thorpe, can you check if anything is wrong with the 
> service-level RCO?
> 
> bhuvnesh chaudhary wrote:
> Thank you Jayush, will go ahead with the change.

When I added the RCO change for putting it at the service level, I only tested 
with services defined in the stack not having inherited the RCO from 
common-services.  I'll look into it some more and see if I can figure out what 
the problem might be.  Sorry was on vacation for a week.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44687/#review123118
---


On March 11, 2016, 9:50 p.m., bhuvnesh chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44687/
> ---
> 
> (Updated March 11, 2016, 9:50 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, 
> Matt, Oleksandr Diachenko, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-15381
> https://issues.apache.org/jira/browse/AMBARI-15381
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Update RCO to ensure that PXF is started after HDFS. 
> PXF connects to namenode during startup and retries couple of times until 
> namenode has come up.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json 
> b28f2a9 
> 
> Diff: https://reviews.apache.org/r/44687/diff/
> 
> 
> Testing
> ---
> 
> yes. manually.
> 
> 
> Thanks,
> 
> bhuvnesh chaudhary
> 
>

Review Request 44956: Topology host info is not cleared when a host is removed


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44956/
---

Review request for Ambari, Alejandro Fernandez, Laszlo Puskas, Oliver Szabo, 
Sandor Magyari, Sumit Mohanty, and Sebastian Toader.


Bugs: AMBARI-15457
https://issues.apache.org/jira/browse/AMBARI-15457


Repository: ambari


Description
---

When a host is removed, a record is left in the topology_host_info table. As a 
result re-adding the same host is not possible, due to invalid topology.
Host removal should remove the corresponding record from topology_host_info as 
well.

A new column with a foreign key to hosts table is added. In addition 
ClusterTopologyImpl was statefull and hostGroupInfoMap was not updated when a 
host was removed.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
 6101ee8 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/TopologyHostInfoDAO.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/TopologyHostInfoEntity.java
 a4f251a 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
 46dd49d 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopology.java
 4e178c0 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java
 003539c 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupInfo.java
 4847a11 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedState.java
 dbf6735 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedStateImpl.java
 b878955 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java
 41f3150 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
 8267d5d 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql dee397f 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 253b5d7 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 832d1bc 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
6c8a09d 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a6acb74 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 2957c5c 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
 3e43ef1 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostResourceProviderTest.java
 69c570e 

Diff: https://reviews.apache.org/r/44956/diff/


Testing
---

Tests ran without failures. (2016-03-17 13:30)


Thanks,

Daniel Gergely

Re: Review Request 45012: Recommendation and validation logic for HAWQ hdfs-client.xml output.replace-datanode-on-failure property


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45012/
---

(Updated March 18, 2016, 7:34 p.m.)


Review request for Ambari, Alejandro Fernandez, bhuvnesh chaudhary, Goutam 
Tadi, jun aoki, Jayush Luniya, Lav Jain, and Sumit Mohanty.


Changes
---

changed to lower-case booleans, updated some descriptions


Bugs: AMBARI-15499
https://issues.apache.org/jira/browse/AMBARI-15499


Repository: ambari


Description
---

Recommendation and validation logic for HAWQ hdfs-client.xml 
output.replace-datanode-on-failure property


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
 93ad3fe 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hdfs-client.xml
 c2a756a 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
c0bc4e2 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 4f0b4b7 

Diff: https://reviews.apache.org/r/45012/diff/


Testing
---

manual

--
Ran 253 tests in 7.111s

OK
--
Total run:927
Total errors:0
Total failures:0
OK


Thanks,

Alexander Denissov

Re: Review Request 44886: AMBARI-15437 : Handle changing symlinks of ranger admin binaries for upgrade scenarios

2016-03-19 Thread Gautam Borad



> On March 16, 2016, 12:24 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py,
> >  lines 128-131
> > 
> >
> > Do you need to:
> > 
> > - Check for the existance of these links first before removing/creating 
> > them?
> > - Check a specific version of the stack that you are upgrading from/to?

1) No, if it already exists, we delete it. The Link() does not throw error, if 
it does not exists in the first place.
2) /usr/bin/ranger-admin is used from stack 2.3 onwards (Also, please note from 
2.3 stack onwards, Ranger uses setup_ranger_xml.py)


- Gautam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44886/#review123860
---


On March 16, 2016, 9:15 a.m., Gautam Borad wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44886/
> ---
> 
> (Updated March 16, 2016, 9:15 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, Jonathan 
> Hurley, Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: AMBARI-15437
> https://issues.apache.org/jira/browse/AMBARI-15437
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> For Ambari stacks for Ranger: 
> We need to take care of handling symlinks to point to current version of HDP, 
> to avoid confusion.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
>  6394f51 
> 
> Diff: https://reviews.apache.org/r/44886/diff/
> 
> 
> Testing
> ---
> 
> Tested EU and RU with Ranger installed.
> Testing steps:
> 1) Installed Ranger with HDP-2.2.x
> 2) Upgrade Ranger to HDP-2.3.x
> 3) Again Upgrade to HDP-2.4.x
> 
> Ran 277 tests in 7.293s
> 
> OK
> --
> Total run:941
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Gautam Borad
> 
>

Re: Review Request 44958: Increase Ambari Server Perm gen default value


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44958/#review124172
---


Ship it!




Ship It!

- Robert Levas


On March 18, 2016, 4:39 a.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44958/
> ---
> 
> (Updated March 18, 2016, 4:39 a.m.)
> 
> 
> Review request for Ambari, Laszlo Puskas, Oliver Szabo, Robert Levas, Sandor 
> Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-15465
> https://issues.apache.org/jira/browse/AMBARI-15465
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Increased permGen size to 128m, also added **-XX:+CMSPermGenSweepingEnabled** 
> and **-XX:+CMSClassUnloadingEnabled** to allow jvm to keep permGen space 
> clean.
> 
> 
> Diffs
> -
> 
>   ambari-server/conf/unix/ambari-env.sh e9fdcdd 
>   ambari-server/src/main/python/ambari_server_main.py 00e084b 
> 
> Diff: https://reviews.apache.org/r/44958/diff/
> 
> 
> Testing
> ---
> 
> Manual testing:
> [root@c6401 sbin]# ps -ef | grep ambari-server
> root 20769 1 27 14:47 pts/000:00:24 
> //usr/jdk64/jdk1.8.0_60/bin/java -server -XX:NewRatio=3 
> -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit 
> -XX:CMSInitiatingOccupancyFraction=60 **-XX:+CMSPermGenSweepingEnabled 
> -XX:+CMSClassUnloadingEnabled** -Dsun.zip.disableMemoryMapping=true -Xms512m 
> -Xmx2048m **-XX:MaxPermSize=128m** 
> -Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
> -Djava.security.krb5.conf=/etc/krb5.conf 
> -Djavax.security.auth.useSubjectCredsOnly=false -Xms512m -Xmx2048m 
> -Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
> -Djava.security.krb5.conf=/etc/krb5.conf 
> -Djavax.security.auth.useSubjectCredsOnly=false -cp 
> //etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar
>  org.apache.ambari.server.controller.AmbariServer
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Re: Review Request 44725: After exporting blueprint from ranger enabled cluster ranger.service.https.attrib.keystore.pass is exported

2016-03-19 Thread Robert Nettleton

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44725/#review124195
---

Thanks for providing this patch, but I believe an alternate approach should be 
explored. 

Please see the issue I've raised below. 

Rather than replacing the existing filter, I'd recommend implementing an 
additional filter that can be registered to handle this special case.

I also wanted to mention that the "property-type" of a given property is 
accessible from the BlueprintConfigurationProcessor.

The following class encapsulates the stack metadata for a given stack:

org.apache.ambari.server.controller.internal.Stack

That class contains a convenience method to determine if a property is a 
password:

org.apache.ambari.server.controller.internal.Stack#isPasswordProperty

It might be useful to implement a new filter that just checks the 
isPasswordProperty() method, and filters based on that.  This filter should be 
implemented in addition to the existing name-based filter, and not necessarily 
as a replacement for it. 

This special handling code is necessary, due to the fact that the Ambari Stack 
metadata does not provide a way to model "conditional properties", such as the 
Ranger password properties mentioned here, that are only required in an HTTPS 
deployment scenario. 

Thanks.

ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
 (line 2663)

I don't think this approach will work generally. 

I've double-checked the code, and it looks like the SecretReference code 
does not necessarily always update the configuration prior to a Blueprint 
export attempt.  

The problem is that the definition of a "password" is somewhat vague 
sometimes in the Ambari Stacks.  There are several types of properties that 
qualify as passwords:

1. Properties marked with the "password" type, and are required as input 
prior to cluster creation.  These are automatically filtered out of an export. 
2. Properties marked with the "password" type, but not marked as required 
input.  These usually have default values as well, such as the Ranger password 
properties that are only valid in an HTTPS deployment. These are filtered out 
by the current PasswordPropertyFilter. 
3. Any properties added to the stack that may be passwords, but are 
incorrectly marked as not being passwords in the metadata. 

The BlueprintConfigurationProcessor must make a best effort to handle all 
these types, and that is why the Filter pattern was adopted a few releases ago.

- Robert Nettleton

On March 17, 2016, 6:52 p.m., Amruta Borkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44725/
> ---
> 
> (Updated March 17, 2016, 6:52 p.m.)
> 
> 
> Review request for Ambari, Di Li and Robert Nettleton.
> 
> 
> Bugs: AMBARI-15338
> https://issues.apache.org/jira/browse/AMBARI-15338
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After exporting blueprint from ranger enabled cluster 
> ranger.service.https.attrib.keystore.pass is exported.
> Which needs to be removed before using the same blueprint to create another 
> cluster
> Error Show when used same blueprint:
> { "status" : 400, "message" : "Blueprint configuration validation failed: 
> Secret references are not allowed in blueprints, replace following properties 
> with real passwords:\n Config:ranger-admin-site 
> Property:ranger.service.https.attrib.keystore.pass\n" }
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  f5e7578 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  68d5755 
> 
> Diff: https://reviews.apache.org/r/44725/diff/
> 
> 
> Testing
> ---
> 
> Modified test cases to test for if the properties that end with "pass" are 
> getting filtered. Other properties which have 'pass' else where in the name 
> will not get filtered.
> 
> 
> Thanks,
> 
> Amruta Borkar
> 
>

Re: Review Request 44987: Editing tmp dir from single to multiple dir doesn't trigger the backend change


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44987/#review124306
---




ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/common.py
 (line 57)


with hawq you don't call master directory for segment hawq_segment_dir



ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/common.py
 (line 67)


probably should have stripped of spaces before creating the dir.


- bhuvnesh chaudhary


On March 18, 2016, 10:17 p.m., jun aoki wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44987/
> ---
> 
> (Updated March 18, 2016, 10:17 p.m.)
> 
> 
> Review request for Ambari.
> 
> 
> Bugs: AMBARI-15475
> https://issues.apache.org/jira/browse/AMBARI-15475
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Modified "HAWQ Segment Temp Directory” from one [/d1/tmp/hawqsegment] to many 
> [/d1/tmp/hawqsegment,/d2/tmp/hawqsegment,/d3/tmp/hawqsegment,/d4/tmp/hawqsegment,/d5/tmp/hawqsegment]
> and the change is not propagated to segments and the folders are not created.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/common.py
>  d62332d 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/hawqsegment.py
>  0a597b6 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/master_helper.py
>  9482b35 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/params.py
>  1bdc5aa 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/utils.py
>  dc6c203 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqmaster.py 14b5a5f 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqsegment.py 6ca83b7 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqstandby.py eaba496 
> 
> Diff: https://reviews.apache.org/r/44987/diff/
> 
> 
> Testing
> ---
> 
> manually tested.
> 
> test_configure_default (test_hawqsegment.TestHawqSegment) ... ok
> test_install_default (test_hawqsegment.TestHawqSegment) ... ok
> test_start_default (test_hawqsegment.TestHawqSegment) ... ok
> test_stop_default (test_hawqsegment.TestHawqSegment) ... ok
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_configure_default (test_hawqstandby.TestHawqStandby) ... ok
> test_install_default (test_hawqstandby.TestHawqStandby) ... ok
> test_start_default (test_hawqstandby.TestHawqStandby) ... 2016-03-17 
> 15:33:42,969 - Standby host c6401.ambari.apache.org initialized
> ok
> test_stop_default (test_hawqstandby.TestHawqStandby) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_start_default (test_hawqmaster.TestHawqMaster) ... 2016-03-17 
> 15:33:43,025 - Active master c6401.ambari.apache.org initialized
> ok
> test_start_localmaster (test_hawqmaster.TestHawqMaster) ... 2016-03-17 
> 15:33:43,049 - Master c6401.ambari.apache.org started
> ok
> test_stop_default (test_hawqmaster.TestHawqMaster) ... ok
> 
> 
> Thanks,
> 
> jun aoki
> 
>

Re: Review Request 44958: Increase Ambari Server Perm gen default value



> On márc. 17, 2016, 4:37 du, Robert Levas wrote:
> > ambari-server/src/main/python/ambari_server_main.py, line 89
> > 
> >
> > I am not familiar with these options, but it apears like no one really 
> > knows what they do.  See https://dzone.com/articles/busting-permgen-myths 
> > for an example. 
> > 
> > Do we really think that this is necessary?  Do we know how this will 
> > affect performance?

These options are for JVM, it can be checked in the jvm implementors webpage. 
For example IBM does not use permgen space at all, but oracle does. The first 
option has not effect from java 7, but it was useful in java 6 and before, so I 
removed that one.
However -XX:+CMSClassUnloadingEnabled is useful. When classes are loaded 
(statically or dynamically) they are stored in permgen space and remains there 
forever (well, almost...). Since we use CMS (-XX:+UseConcMarkSweepGC flag is 
already defined), we can allow GC to unload unused classes. As a result it can 
increase the free space in permGen. In normal case when permGen space is full, 
garbage collector initiates a STW (stop-the-world) action to clean permgen 
space.
Short description here: 
https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent
Oracle tutorial here: 
http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html

>From Java 8 as I know there is no permGen space, so this issue relates only to 
>java 7.

So I removed -XX:+CMSPermGenSweepingEnabled, but I kept 
-XX:+CMSClassUnloadingEnabled. Performance wise you should not experience any 
difference.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44958/#review124042
---


On márc. 17, 2016, 2:57 du, Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44958/
> ---
> 
> (Updated márc. 17, 2016, 2:57 du)
> 
> 
> Review request for Ambari, Laszlo Puskas, Oliver Szabo, Robert Levas, Sandor 
> Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-15465
> https://issues.apache.org/jira/browse/AMBARI-15465
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Increased permGen size to 128m, also added **-XX:+CMSPermGenSweepingEnabled** 
> and **-XX:+CMSClassUnloadingEnabled** to allow jvm to keep permGen space 
> clean.
> 
> 
> Diffs
> -
> 
>   ambari-server/conf/unix/ambari-env.sh e9fdcdd 
>   ambari-server/src/main/python/ambari_server_main.py 00e084b 
> 
> Diff: https://reviews.apache.org/r/44958/diff/
> 
> 
> Testing
> ---
> 
> Manual testing:
> [root@c6401 sbin]# ps -ef | grep ambari-server
> root 20769 1 27 14:47 pts/000:00:24 
> //usr/jdk64/jdk1.8.0_60/bin/java -server -XX:NewRatio=3 
> -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit 
> -XX:CMSInitiatingOccupancyFraction=60 **-XX:+CMSPermGenSweepingEnabled 
> -XX:+CMSClassUnloadingEnabled** -Dsun.zip.disableMemoryMapping=true -Xms512m 
> -Xmx2048m **-XX:MaxPermSize=128m** 
> -Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
> -Djava.security.krb5.conf=/etc/krb5.conf 
> -Djavax.security.auth.useSubjectCredsOnly=false -Xms512m -Xmx2048m 
> -Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
> -Djava.security.krb5.conf=/etc/krb5.conf 
> -Djavax.security.auth.useSubjectCredsOnly=false -cp 
> //etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar
>  org.apache.ambari.server.controller.AmbariServer
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Re: Review Request 44931: Use Version Definition value for package-version when installing


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44931/
---

(Updated March 16, 2016, 6:08 p.m.)


Review request for Ambari, Alejandro Fernandez and Jonathan Hurley.


Bugs: AMBARI-15447
https://issues.apache.org/jira/browse/AMBARI-15447


Repository: ambari


Description
---

The version defintion file defines a package version that should be used when 
installing packages.  The current calculation cannot be considered reliable, 
but should stay as a contingency for when NOT using a version definition.


Diffs
-

  ambari-common/src/main/python/resource_management/libraries/script/script.py 
a8098a0 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 402a338 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 e32e2f9 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
 07e62b3 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostStackVersionResourceProvider.java
 1cd9c0a 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
 3398709 
  
ambari-server/src/main/java/org/apache/ambari/server/state/repository/Release.java
 6bcedf5 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProviderTest.java
 dea83a1 
  ambari-server/src/test/python/custom_actions/TestInstallPackages.py f022c80 
  ambari-server/src/test/resources/hbase_version_test.xml 9df07ed 

Diff: https://reviews.apache.org/r/44931/diff/


Testing (updated)
---

Manual.  Automated:

Tests run: 3961, Failures: 0, Errors: 0, Skipped: 33

[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 43:00.985s
[INFO] Finished at: Wed Mar 16 17:50:32 EDT 2016
[INFO] Final Memory: 35M/776M
[INFO] 


Thanks,

Nate Cole

Re: Review Request 44962: file_system get_mount_point_for_dir works incorrect

2016-03-19 Thread Dmytro Sen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44962/#review124045
---


Ship it!




Ship It!

- Dmytro Sen


On Март 17, 2016, 4:03 п.п., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44962/
> ---
> 
> (Updated Март 17, 2016, 4:03 п.п.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Dmytro Sen.
> 
> 
> Bugs: AMBARI-15468
> https://issues.apache.org/jira/browse/AMBARI-15468
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> **Case 1**:
> 
> 
> [root@c6403 ~]# mount
> /dev/mapper/VolGroup-lv_root on / type ext4 (rw)
> /newFs on /newHdfsDir type ext3 (rw,loop=/dev/loop1)
> 
> [root@c6403 ~]# python
> >>> from resource_management import *
> >>> from resource_management.libraries.functions import file_system
> >>> with Environment() as env:
> ...   print file_system.get_mount_point_for_dir('/newHdfsDir/some_dir')
> ...   print file_system.get_mount_point_for_dir('/newHdfsDir')
> ... 
> 2016-03-17 13:00:33,640 - Mount point for directory /newHdfsDir/some_dir 
> is /
> 2016-03-17 13:00:33,640 - Mount point for directory /newHdfsDir is /
> 
> 
> It should be /newHdfsDir, not /
> 
> **Reason**   
> /newHdfsDir and / have the same count of os separators.
> 
> **Case 2**:
> 
> 
> [root@c6403 ~]# mount
> /dev/mapper/VolGroup-lv_root on / type ext4 (rw)
> /newFs on /a/a type ext3 (rw,loop=/dev/loop2)
> /newFs on /a/a1 type ext3 (rw,loop=/dev/loop3)
> 
> [root@c6403 ~]# python
> >>> from resource_management import *
> >>> from resource_management.libraries.functions import file_system
> >>> with Environment() as env:
> >>> from resource_management import *
> >>> from resource_management.libraries.functions import file_system
> >>> with Environment() as env:
> ...   print file_system.get_mount_point_for_dir('/a/a1/some_dir')
> ...   print file_system.get_mount_point_for_dir('/a/a1')
> ...
> 2016-03-17 13:07:38,899 - Mount point for directory /a/a1/some_dir is /a/a
> 2016-03-17 13:07:38,900 - Mount point for directory /a/a1 is /a/a
> 
> 
> It should be /a/a1, not /a/a
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/test/python/resource_management/TestFileSystem.py 4e0eb63 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/file_system.py
>  39b86dd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 6132c2c 
> 
> Diff: https://reviews.apache.org/r/44962/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 44923: Alerts: Allow alerts to emit a structured response

2016-03-19 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44923/
---

(Updated Березень 17, 2016, 8:11 після полудня)


Review request for Ambari, Dmytro Sen and Jonathan Hurley.


Bugs: AMBARI-15445
https://issues.apache.org/jira/browse/AMBARI-15445


Repository: ambari


Description
---

For some alerts, having the ability to include a link in the response would be 
incredibly beneficial.  Having a structured response format would allow alert 
creators to add links that operators could click on to help give more context 
around an alert, or point to a remediation step.  Something like:

{code}
{
  "response": "There were more than 5 soft limits reached in the past 10 
minutes.  Please click the link to see the log file.",
  "href": "http://some.server/other/stuff";
}
{code}


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertDefinitionResourceProvider.java
 0f73ec6 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertDefinitionEntity.java
 9e21bec 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertDefinitionEntity_.java
 d2d4cf3 
  
ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinition.java
 9fff5f2 
  
ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinitionFactory.java
 3b4f5fc 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 a803f73 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql a85202d 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 9b4810c 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql cc3d197 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 07c786d 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
ab6dc93 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 8e91fde 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 440ca44 

Diff: https://reviews.apache.org/r/44923/diff/


Testing
---

Tests will be added after patch draft review


Thanks,

Vitalyi Brodetskyi

Re: Review Request 44880: AMBARI-15436 Supportability: Create 'Reset UI' button in Ambari's experimental page

2016-03-19 Thread Zhe (Joe) Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44880/
---

(Updated March 17, 2016, 8:52 p.m.)


Review request for Ambari, Jaimin Jetly, Richard Zang, Srimanth Gunturi, Xi 
Wang, and Yusaku Sako.


Bugs: AMBARI-15436
https://issues.apache.org/jira/browse/AMBARI-15436


Repository: ambari


Description
---

Currently Ambari UI persists it's current state on the server. This helps when 
a different user logs in, they end up in the same UI. Sometimes however, UI's 
state persisted on server does not sync with cluster state, thereby putting UI 
in a broken state with no recourse for the stuck user.

Ambari UI should provide a _Reset UI_ button in the experimental page which 
resets the UI state on server.


Diffs (updated)
-

  ambari-web/app/controllers/experimental.js 9fa6da5 
  ambari-web/app/mappers/server_data_mapper.js 1e08161 
  ambari-web/app/messages.js 9289fc2 
  ambari-web/app/templates/experimental.hbs 2873679 

Diff: https://reviews.apache.org/r/44880/diff/


Testing
---

Local ambari-web test passed.
24578 tests complete (23 seconds)
145 tests pending
Manual testing done.


Thanks,

Zhe (Joe) Wang

Re: Review Request 44959: Enable Ambari build process to automatically switch repo base urls in the repo files

2016-03-19 Thread Sumit Mohanty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44959/#review124020
---


Ship it!




Ship It!

- Sumit Mohanty


On March 17, 2016, 2:49 p.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44959/
> ---
> 
> (Updated March 17, 2016, 2:49 p.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-15467
> https://issues.apache.org/jira/browse/AMBARI-15467
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Today, there is a manual step, at the end of each release to update the repo
> baseurls from internal dev repo to external public repo. This is error prone
> and easy to miss.
> 
> Ambari build process should be modified to allow RE to specify the public repo
> urls as the baseUrl for the repos.
> 
> 
> Diffs
> -
> 
>   ambari-common/src/main/python/urlinfo_processor/urlinfo_processor.py 
> PRE-CREATION 
>   ambari-server/pom.xml 5bb3b98 
> 
> Diff: https://reviews.apache.org/r/44959/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Review Request 44933: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.

2016-03-19 Thread Juanjo Marron


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/
---

Review request for Ambari, Alejandro Fernandez and Jayush Luniya.


Bugs: AMBARI-14435
https://issues.apache.org/jira/browse/AMBARI-14435


Repository: ambari


Description
---

Apply the  stack featurization prototype  detailed on AMBARI-13364 to ZOOKEEPER 
service.
This patch removes the hardcoded stack versions in ZK common-services code 
(centralized version and parameterized stack configurations will be faced in 
other JIRAS)
stack_features is now a property in .json format in 
configuration/cluster-env.xml for each stack version.
With the structure:
{
"stack_features": [
{ "name": "feature1", "description" : "Feature1 support", "min_version" : 
"Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
, 
...
]
}
where min_version/max_version are optional constraints.
New resource_management/libraries/functions/stack_fetaures.py has been 
introduced to parse the json file and called from service code to check if the 
stack supports the required feature.


Diffs
-

  
ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
 9ff9125 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py
 d18e4d7 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py
 5712ce4 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py
 25ace24 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py
 d5b6898 
  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py
 6fe0772 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 
3fb82e9 
  ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 
f1fa4de 
  ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml 
PRE-CREATION 

Diff: https://reviews.apache.org/r/44933/diff/


Testing
---

Zookeeper fresh installation has been tested using HDP stack 2.3 and 
AMBARI-13364 branch code.


Thanks,

Juanjo  Marron

Re: Review Request 44941: Report AMS / Grafana Per Disk Metrics and other changes

2016-03-19 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44941/
---

(Updated March 17, 2016, 1:09 a.m.)


Review request for Ambari, Aravindan Vijayan, Dmytro Sen, and Sumit Mohanty.


Changes
---

- Added swap_in and swap_out
- Fixed memory used to reflect (used - cached)


Bugs: AMBARI-15448
https://issues.apache.org/jira/browse/AMBARI-15448


Repository: ambari


Description
---

We need to collect and expose per-disk metrics from AMS for iostats counters.
We can sort the disk names alphabetically, and name the metrics like _1, _2, 
etc.

Plus from psutils:
busy_time: (Linux, FreeBSD) time spent doing actual I/Os (in milliseconds)
read_merged_count (Linux): number of merged reads (see iostat doc)
write_merged_count (Linux): number of merged writes (see iostats doc)

_Additional fixes_:
- Combined_disk_total metric seems to be wrong : it is a sum() of all disk 
percentages from different disks
- Missing ATS and JHS metrics due to wrong config item in 
hadoop-metrics2.properties file
- Report number of disks


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
 a053955 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/controller.py
 c04a61b 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/host_info.py 
2e8c442 
  
ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/metric_collector.py
 c28fd03 
  
ambari-metrics/ambari-metrics-host-monitoring/src/test/python/core/TestHostInfo.py
 f07b573 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_grafana_util.py
 e5020a5 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
 0b0932a 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2
 47b504f 

Diff: https://reviews.apache.org/r/44941/diff/


Testing
---

Manually verified.

--
Ran 12 tests in 0.020s

OK
--
Total run:12
Total errors:0
Total failures:0
OK


Thanks,

Sid Wagle

Review Request 45054: AMBARI-15443:Make Host bulk command menu item list stack driven instead of a hardcoded list in UI code

2016-03-19 Thread Di Li


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45054/
---

Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, and Yusaku Sako.


Bugs: AMBARI-15443
https://issues.apache.org/jira/browse/AMBARI-15443


Repository: ambari


Description
---

The items described here are the ones on the Hosts tab, in the Actions 
drop-down list where the UI shows entries such as All Hosts. If user mouses 
over the All Hosts, it shows a sub-list including Hosts and slave components.

The slave component items are hardcoded in hosts_table_menu_view.js as shown 
below. This jira is to put this info into each service's metainfo.xml so that 
the slave component items can be stack driven.

components: function () {
var serviceNames = App.Service.find().mapProperty('serviceName');
var menuItems = [
O.create(
{ serviceName: 'HDFS', componentName: 'DATANODE', masterComponentName: 
'NAMENODE', componentNameFormatted: 
Em.I18n.t('dashboard.services.hdfs.datanodes') }

),
O.create(
{ serviceName: 'YARN', componentName: 'NODEMANAGER', masterComponentName: 
'RESOURCEMANAGER', componentNameFormatted: 
Em.I18n.t('dashboard.services.yarn.nodeManagers') }

),
O.create(
{ serviceName: 'HBASE', componentName: 'HBASE_REGIONSERVER', 
masterComponentName: 'HBASE_MASTER', componentNameFormatted: 
Em.I18n.t('dashboard.services.hbase.regionServers') }

),
O.create(
{ serviceName: 'STORM', componentName: 'SUPERVISOR', masterComponentName: 
'SUPERVISOR', componentNameFormatted: 
Em.I18n.t('dashboard.services.storm.supervisors') }

)];

return menuItems.filter(function (item)
{ return serviceNames.contains(item.serviceName); }

);
}.property(),


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceComponentResponse.java
 cfd4e7b 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceComponentResourceProvider.java
 70945ba 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ComponentModule.java 
a122dc6 
  
ambari-server/src/main/java/org/apache/ambari/server/state/BulkCommandDefinition.java
 PRE-CREATION 
  ambari-server/src/main/java/org/apache/ambari/server/state/ComponentInfo.java 
770ee5c 
  ambari-server/src/main/resources/common-services/HAWQ/2.0.0/metainfo.xml 
e35b7d8 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/metainfo.xml 
057e126 
  ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/metainfo.xml 
75d3bea 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/metainfo.xml 
afe27ec 
  ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/metainfo.xml 
804374a 
  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/metainfo.xml 
0f71585 
  ambari-server/src/main/resources/properties.json 01c15f2 
  
ambari-server/src/test/java/org/apache/ambari/server/stack/ComponentModuleTest.java
 2f84f04 
  ambari-web/app/mappers/stack_service_mapper.js 8a65055 
  ambari-web/app/models/stack_service_component.js 26ff1b8 
  ambari-web/app/views/main/host/hosts_table_menu_view.js e75b643 
  ambari-web/test/mappers/stack_service_mapper_test.js 4bc36fe 

Diff: https://reviews.apache.org/r/45054/diff/


Testing
---

unit test
patch a trunk cluster with code change, verify the host bulk command list shown 
for DataNode, HBase Region server, storm supervisor, etc.


Thanks,

Di Li

Re: Review Request 44956: Topology host info is not cleared when a host is removed

2016-03-19 Thread Sebastian Toader


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44956/#review124168
---


Ship it!




Ship It!

- Sebastian Toader


On March 18, 2016, 10:27 a.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44956/
> ---
> 
> (Updated March 18, 2016, 10:27 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Laszlo Puskas, Oliver Szabo, 
> Sandor Magyari, Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-15457
> https://issues.apache.org/jira/browse/AMBARI-15457
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a host is removed, a record is left in the topology_host_info table. As 
> a result re-adding the same host is not possible, due to invalid topology.
> Host removal should remove the corresponding record from topology_host_info 
> as well.
> 
> A new column with a foreign key to hosts table is added. In addition 
> ClusterTopologyImpl was statefull and hostGroupInfoMap was not updated when a 
> host was removed.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
>  6101ee8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/TopologyHostInfoDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/TopologyHostInfoEntity.java
>  a4f251a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
>  46dd49d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopology.java
>  4e178c0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java
>  003539c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupInfo.java
>  4847a11 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedState.java
>  dbf6735 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedStateImpl.java
>  b878955 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java
>  41f3150 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
>  8267d5d 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql dee397f 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 253b5d7 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 832d1bc 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> 6c8a09d 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a6acb74 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 2957c5c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  3e43ef1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostResourceProviderTest.java
>  69c570e 
> 
> Diff: https://reviews.apache.org/r/44956/diff/
> 
> 
> Testing
> ---
> 
> Tests ran without failures. (2016-03-17 13:30)
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Review Request 44926: [DRAFT] Auto-retry on failure during RU/EU

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44926/
---

Review request for Ambari, Jonathan Hurley and Nate Cole.

Bugs: AMBARI-15446
https://issues.apache.org/jira/browse/AMBARI-15446

Repository: ambari

Description
---

When a failure occurs during RU/EU and the task transitions to HOLDING_FAILED
or HOLDING_TIMEDOUT, want Ambari to automatically retry up to up to x mins.
This is useful when a host goes down as Ambari is running a task on it.
ambari.properties will have 1 new parameter. E.g,.
stack-upgrade.max_retry_timeout_mins=15 (by default, will not be present)
If Ambari Server is restarted, it should be able to recover.
Today, Action Scheduler increases the attempt_count whenever a task is retried,
but it requires resetting the start_time to -1. Because of this, we cannot rely
on the start_time property to know when to timeout after several retries.

For the implementation, will add another thread to Ambari that will monitor
failed tasks only during active RU/EU and change the status back to PENDING so
that Action Scheduler can reschedule it.
Luckily, any tasks in HOLDING_TIMEDOUT and HOLDING_FAILED states are blocking,
so no other stages are allowed to proceed.
In order to know when a task was first started, will add a new property to
host_role_command table called original_start_time.

For the agents, we need to ensure that they always write out a response. On the
first heartbeat, it should send the status of its last command so we know it
failed and Ambari can retry.

Diffs
-

ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
3a80803

ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
PRE-CREATION

ambari-server/src/main/java/org/apache/ambari/server/checks/PreviousUpgradeCompleted.java
3a4467f

ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterVersionDAO.java
1bcca60

ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
f5b1cb4

ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterVersionEntity.java
f1867b4

ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java
19f0602
ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java
ed3c772

ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
1c7ff61

ambari-server/src/main/java/org/apache/ambari/server/topology/LogicalRequest.java
82edbcf

Diff: https://reviews.apache.org/r/44926/diff/

Testing
---

Verified on a live cluster.

TODO: Still need to make more changes to the implementation, add the config,
switch to gauva service, add a column, and add unit tests.

Thanks,

Alejandro Fernandez

Re: Review Request 45035: Restarting HDFS Before Upgrade Finalizing Does Not Supply the rollingUpgrade Flag



> On March 18, 2016, 1:50 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java,
> >  line 453
> > 
> >
> > I think adding a new column is overkill. If EU is suspended while 
> > stopping services, then HDFS has not yet been prepared or restarted with 
> > the -rollingUpgrade option.
> > 
> > I wonder if there's a way to detect on the python side that NameNode is 
> > in the middle of an upgrade and hence should be restarted with that flag.
> > Could even use a marker file that gets deleted after Finalize.

I don't think it is; it's useful for querying and allows us to deliver this 
state information via commands to any service that needs it; not just HDFS. 

The marker file is the hack approach - easily can cause problems.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45035/#review124232
---


On March 18, 2016, 11:20 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45035/
> ---
> 
> (Updated March 18, 2016, 11:20 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.
> 
> 
> Bugs: AMBARI-15482
> https://issues.apache.org/jira/browse/AMBARI-15482
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During an upgrade, the HDFS NameNode(s) are restarted with the 
> {{-rollingUpgrade}} flag. However, it's possible to get to the end of an 
> upgrade and decide to "Finalize Later".
> 
> This allows the cluster to run in the upgraded state before committing to the 
> upgrade. Full cluster control is returned via the Ambari web client.
> 
> Administrators can then decide to restart a NameNode. Upon restarting the 
> NameNode, it will produce an error that it was not started with the 
> {{rollingUpgrade}} flag. 
> 
> It seems that as long as an upgrade has not been finalized, the NameNode(s) 
> must be started with the {{rollingUpgrade}} to allow them to function 
> properly. 
> 
> STR:
> - Perform a rolling upgrade from HDP 2.2 to 2.3 (or 2.3 to 2.4); as long as 
> there is a major version change.
> - Before finalization, say "Finalize Later". All services should be up and 
> green.
> - Now restart a NameNode
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  9ea541e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
>  4ef215c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  303f3a4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  07061e1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
>  3f1a52b 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> ed3c772 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  1c7ff61 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
>  84bb9f3 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql a85202d 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 9b4810c 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql cc3d197 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 07c786d 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> ab6dc93 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 8e91fde 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 440ca44 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
>  e4c8c9c 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  02905ec 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
>  905802f 
>   ambari-web/app/controllers/main/admin/stack_and_upgrade_controller.js 
> 2dceccc 
>   ambari-web/app/utils/ajax/ajax.js 29d0715 
> 
> Diff: https://reviews.apache.org/r/45035/diff/
> 
> 
> Testing
> ---
> 
> Pending...
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 44956: Topology host info is not cleared when a host is removed



> On márc. 17, 2016, 6:23 du, Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java,
> >  line 145
> > 
> >
> > May need to ensure this is idempotent.

Implementation of addFKConstraint checks if the FK exists before executing the 
addition.


- Daniel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44956/#review124068
---


On márc. 17, 2016, 12:59 du, Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44956/
> ---
> 
> (Updated márc. 17, 2016, 12:59 du)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Laszlo Puskas, Oliver Szabo, 
> Sandor Magyari, Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-15457
> https://issues.apache.org/jira/browse/AMBARI-15457
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a host is removed, a record is left in the topology_host_info table. As 
> a result re-adding the same host is not possible, due to invalid topology.
> Host removal should remove the corresponding record from topology_host_info 
> as well.
> 
> A new column with a foreign key to hosts table is added. In addition 
> ClusterTopologyImpl was statefull and hostGroupInfoMap was not updated when a 
> host was removed.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
>  6101ee8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/TopologyHostInfoDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/TopologyHostInfoEntity.java
>  a4f251a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
>  46dd49d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopology.java
>  4e178c0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java
>  003539c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupInfo.java
>  4847a11 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedState.java
>  dbf6735 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedStateImpl.java
>  b878955 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java
>  41f3150 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
>  8267d5d 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql dee397f 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 253b5d7 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 832d1bc 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> 6c8a09d 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a6acb74 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 2957c5c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  3e43ef1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostResourceProviderTest.java
>  69c570e 
> 
> Diff: https://reviews.apache.org/r/44956/diff/
> 
> 
> Testing
> ---
> 
> Tests ran without failures. (2016-03-17 13:30)
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Review Request 45045: enable ssl for ambari does not update the tez view url in tez-site.xml


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45045/
---

Review request for Ambari and Sumit Mohanty.


Bugs: AMBARI-15484
https://issues.apache.org/jira/browse/AMBARI-15484


Repository: ambari


Description
---

When one enables ssl for ambari we need to update the url set in tez.tez-ui
.history-url.base. We should update that to the new https url or atleast add
that to the enable ssl doc for ambari so the users know they have to update
it.


Diffs
-

  ambari-server/src/main/python/ambari_server/setupHttps.py c9fe421 
  ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 
5e23b8a 
  ambari-server/src/test/python/TestAmbariServer.py a1b9cac 
  ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 9b2da28 

Diff: https://reviews.apache.org/r/45045/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 44880: AMBARI-15436 Supportability: Create 'Reset UI' button in Ambari's experimental page

2016-03-19 Thread Srimanth Gunturi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44880/#review124098
---


Ship it!




Ship It!

- Srimanth Gunturi


On March 17, 2016, 8:52 p.m., Zhe (Joe) Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44880/
> ---
> 
> (Updated March 17, 2016, 8:52 p.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly, Richard Zang, Srimanth Gunturi, Xi 
> Wang, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15436
> https://issues.apache.org/jira/browse/AMBARI-15436
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently Ambari UI persists it's current state on the server. This helps 
> when a different user logs in, they end up in the same UI. Sometimes however, 
> UI's state persisted on server does not sync with cluster state, thereby 
> putting UI in a broken state with no recourse for the stuck user.
> 
> Ambari UI should provide a _Reset UI_ button in the experimental page which 
> resets the UI state on server.
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/experimental.js 9fa6da5 
>   ambari-web/app/mappers/server_data_mapper.js 1e08161 
>   ambari-web/app/messages.js 9289fc2 
>   ambari-web/app/templates/experimental.hbs 2873679 
> 
> Diff: https://reviews.apache.org/r/44880/diff/
> 
> 
> Testing
> ---
> 
> Local ambari-web test passed.
> 24578 tests complete (23 seconds)
> 145 tests pending
> Manual testing done.
> 
> 
> Thanks,
> 
> Zhe (Joe) Wang
> 
>

Re: Review Request 45035: Restarting HDFS Before Upgrade Finalizing Does Not Supply the rollingUpgrade Flag



> On March 18, 2016, 1:50 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java,
> >  line 453
> > 
> >
> > I think adding a new column is overkill. If EU is suspended while 
> > stopping services, then HDFS has not yet been prepared or restarted with 
> > the -rollingUpgrade option.
> > 
> > I wonder if there's a way to detect on the python side that NameNode is 
> > in the middle of an upgrade and hence should be restarted with that flag.
> > Could even use a marker file that gets deleted after Finalize.
> 
> Jonathan Hurley wrote:
> I don't think it is; it's useful for querying and allows us to deliver 
> this state information via commands to any service that needs it; not just 
> HDFS. 
> 
> The marker file is the hack approach - easily can cause problems.

Hmmm - I see what you're saying though. What if HDFS hasn't even started yet. 
And then they pause the upgrade and try to restart NameNode. OK, so how about a 
combination then; we can use a marker file, but we also use the suspended 
nature of the upgrade in combination with it?


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45035/#review124232
---


On March 18, 2016, 11:20 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45035/
> ---
> 
> (Updated March 18, 2016, 11:20 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.
> 
> 
> Bugs: AMBARI-15482
> https://issues.apache.org/jira/browse/AMBARI-15482
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> During an upgrade, the HDFS NameNode(s) are restarted with the 
> {{-rollingUpgrade}} flag. However, it's possible to get to the end of an 
> upgrade and decide to "Finalize Later".
> 
> This allows the cluster to run in the upgraded state before committing to the 
> upgrade. Full cluster control is returned via the Ambari web client.
> 
> Administrators can then decide to restart a NameNode. Upon restarting the 
> NameNode, it will produce an error that it was not started with the 
> {{rollingUpgrade}} flag. 
> 
> It seems that as long as an upgrade has not been finalized, the NameNode(s) 
> must be started with the {{rollingUpgrade}} to allow them to function 
> properly. 
> 
> STR:
> - Perform a rolling upgrade from HDP 2.2 to 2.3 (or 2.3 to 2.4); as long as 
> there is a major version change.
> - Before finalization, say "Finalize Later". All services should be up and 
> green.
> - Now restart a NameNode
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  9ea541e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
>  4ef215c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
>  303f3a4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  07061e1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
>  3f1a52b 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> ed3c772 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  1c7ff61 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
>  84bb9f3 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql a85202d 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 9b4810c 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql cc3d197 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 07c786d 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> ab6dc93 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 8e91fde 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 440ca44 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
>  e4c8c9c 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
>  02905ec 
>   
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
>  905802f 
>   ambari-web/app/controllers/main/admin/stack_and_upgrade_controller.js 
> 2dceccc 
>   ambari-web/app/utils/ajax/ajax.js 29d0715 
> 
> Diff: https://reviews.apache.org/r/45035/diff/
> 
> 
> Testing
> ---
> 
> Pending...
> 
> 
> Thanks,
> 
> Jonathan Hurley
>

Review Request 44958: Increase Ambari Server Perm gen default value


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44958/
---

Review request for Ambari, Laszlo Puskas, Oliver Szabo, Robert Levas, Sandor 
Magyari, and Sebastian Toader.


Bugs: AMBARI-15465
https://issues.apache.org/jira/browse/AMBARI-15465


Repository: ambari


Description
---

Increased permGen size to 128m, also added **-XX:+CMSPermGenSweepingEnabled** 
and **-XX:+CMSClassUnloadingEnabled** to allow jvm to keep permGen space clean.


Diffs
-

  ambari-server/conf/unix/ambari-env.sh e9fdcdd 
  ambari-server/src/main/python/ambari_server_main.py 00e084b 

Diff: https://reviews.apache.org/r/44958/diff/


Testing
---

Manual testing:
[root@c6401 sbin]# ps -ef | grep ambari-server
root 20769 1 27 14:47 pts/000:00:24 
//usr/jdk64/jdk1.8.0_60/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC 
-XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 
**-XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled** 
-Dsun.zip.disableMemoryMapping=true -Xms512m -Xmx2048m **-XX:MaxPermSize=128m** 
-Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
-Djava.security.krb5.conf=/etc/krb5.conf 
-Djavax.security.auth.useSubjectCredsOnly=false -Xms512m -Xmx2048m 
-Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
-Djava.security.krb5.conf=/etc/krb5.conf 
-Djavax.security.auth.useSubjectCredsOnly=false -cp 
//etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar
 org.apache.ambari.server.controller.AmbariServer


Thanks,

Daniel Gergely

Re: Review Request 45012: Recommendation and validation logic for HAWQ hdfs-client.xml output.replace-datanode-on-failure property



> On March 18, 2016, 5:08 p.m., jun aoki wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py, 
> > line 699
> > 
> >
> > do you want to inline this since propertyvalue is used only once.

I prefer to leave it separate for clarity


> On March 18, 2016, 5:08 p.m., jun aoki wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py, 
> > line 984
> > 
> >
> > Do you want to set (constant) variable for a magic number 4?
> > ```
> > REPLACE_DATANODE_ON_FAILURE_THREASHOLD = 4 # my poor naming skill...
> > if numSegments > REPLACE_DATANODE_ON_FAILURE_THREASHOLD and value ... :
> >   ...
> > elif numSegments <= REPLACE_DATANODE_ON_FAILURE_THREASHOLD and value 
> > ... : 
> >   ...
> > 
> > ```
> > 
> > and you can use it on line 694

added a constant, but not for the whole file, but rather within the method. I 
feel that exposing the constant for the whole file is too intrusive.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45012/#review124221
---


On March 18, 2016, 7:47 p.m., Alexander Denissov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45012/
> ---
> 
> (Updated March 18, 2016, 7:47 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, bhuvnesh chaudhary, Goutam 
> Tadi, jun aoki, Jayush Luniya, Lav Jain, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-15499
> https://issues.apache.org/jira/browse/AMBARI-15499
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Recommendation and validation logic for HAWQ hdfs-client.xml 
> output.replace-datanode-on-failure property
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  93ad3fe 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hdfs-client.xml
>  c2a756a 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> c0bc4e2 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 4f0b4b7 
> 
> Diff: https://reviews.apache.org/r/45012/diff/
> 
> 
> Testing
> ---
> 
> manual
> 
> --
> Ran 253 tests in 7.111s
> 
> OK
> --
> Total run:927
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Alexander Denissov
> 
>

Re: Review Request 45030: Replace 0.0.0.0 with hostname in ams-site/timeline.metrics.service.webapp.address


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45030/#review124198
---


Ship it!





ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
 (line 135)


Instead of 'special' can we call this something more meaningful; like 
`BIND_ALL_IP_ADDRESS`?


- Jonathan Hurley


On March 18, 2016, 9:45 a.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45030/
> ---
> 
> (Updated March 18, 2016, 9:45 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15480
> https://issues.apache.org/jira/browse/AMBARI-15480
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> We need collector hostname in configs. 
> Replace 0.0.0.0 with hostname in 
> ams-site/timeline.metrics.service.webapp.address
> Implement changes for blueprints, stack_advisor, upgrade catalog
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  e73e38f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
>  e10585e 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/params.py
>  3b8505b 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  45c1811 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  12c0f25 
>   
> ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/params.py
>  5ec879c 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
>  b517eeb 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/params.py
>  f75d578 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py
>  a022282 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
>  7a023cf 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> ee2007f 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/hooks/before-START/scripts/params.py
>  469e709 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  ef5dc76 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java
>  859513b 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default.json fe92b5b 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_ams_embedded.json 
> 688dbe0 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 
> 4648ffc 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> 9b2da28 
>   
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json
>  68afb68 
>   ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 
> 79c5580 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 
> a7057db 
>   
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json 
> 390f9ce 
>   ambari-server/src/test/python/stacks/2.3/configs/ats_1_5.json 3fe1235 
> 
> Diff: https://reviews.apache.org/r/45030/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Review Request 44960: AMBARI-15466. RBAC: Add dialog popup to explain roles

2016-03-19 Thread Richard Zang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44960/
---

Review request for Ambari and Xi Wang.


Bugs: AMBARI-15466
https://issues.apache.org/jira/browse/AMBARI-15466


Repository: ambari


Description
---

Add dialog popup to show access control list which is based on back-end role & 
authorization data.


Diffs
-

  ambari-admin/src/main/resources/ui/admin-web/app/index.html 6557b55 
  
ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/clusters/ClustersManageAccessCtrl.js
 8f932e1 
  ambari-admin/src/main/resources/ui/admin-web/app/scripts/i18n.config.js 
228be5c 
  ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/Cluster.js 
806247a 
  
ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/RoleDetailsModal.js
 PRE-CREATION 
  ambari-admin/src/main/resources/ui/admin-web/app/styles/main.css b705a90 
  
ambari-admin/src/main/resources/ui/admin-web/app/views/clusters/manageAccess.html
 fd5285e 
  
ambari-admin/src/main/resources/ui/admin-web/app/views/modals/RoleDetailsModal.html
 PRE-CREATION 
  
ambari-admin/src/main/resources/ui/admin-web/test/unit/controllers/clusters/UserAccessListCtrl_test.js
 e63f34e 

Diff: https://reviews.apache.org/r/44960/diff/


Testing
---

Manually tested on live cluster.
All unit tests passed.


Thanks,

Richard Zang

Re: Review Request 44958: Increase Ambari Server Perm gen default value



> On March 17, 2016, 12:37 p.m., Robert Levas wrote:
> > ambari-server/src/main/python/ambari_server_main.py, line 89
> > 
> >
> > I am not familiar with these options, but it apears like no one really 
> > knows what they do.  See https://dzone.com/articles/busting-permgen-myths 
> > for an example. 
> > 
> > Do we really think that this is necessary?  Do we know how this will 
> > affect performance?
> 
> Daniel Gergely wrote:
> These options are for JVM, it can be checked in the jvm implementors 
> webpage. For example IBM does not use permgen space at all, but oracle does. 
> The first option has not effect from java 7, but it was useful in java 6 and 
> before, so I removed that one.
> However -XX:+CMSClassUnloadingEnabled is useful. When classes are loaded 
> (statically or dynamically) they are stored in permgen space and remains 
> there forever (well, almost...). Since we use CMS (-XX:+UseConcMarkSweepGC 
> flag is already defined), we can allow GC to unload unused classes. As a 
> result it can increase the free space in permGen. In normal case when permGen 
> space is full, garbage collector initiates a STW (stop-the-world) action to 
> clean permgen space.
> Short description here: 
> https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent
> Oracle tutorial here: 
> http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html
> 
> From Java 8 as I know there is no permGen space, so this issue relates 
> only to java 7.
> 
> So I removed -XX:+CMSPermGenSweepingEnabled, but I kept 
> -XX:+CMSClassUnloadingEnabled. Performance wise you should not experience any 
> difference.

Thanks for the clarification.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44958/#review124042
---


On March 18, 2016, 4:39 a.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44958/
> ---
> 
> (Updated March 18, 2016, 4:39 a.m.)
> 
> 
> Review request for Ambari, Laszlo Puskas, Oliver Szabo, Robert Levas, Sandor 
> Magyari, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-15465
> https://issues.apache.org/jira/browse/AMBARI-15465
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Increased permGen size to 128m, also added **-XX:+CMSPermGenSweepingEnabled** 
> and **-XX:+CMSClassUnloadingEnabled** to allow jvm to keep permGen space 
> clean.
> 
> 
> Diffs
> -
> 
>   ambari-server/conf/unix/ambari-env.sh e9fdcdd 
>   ambari-server/src/main/python/ambari_server_main.py 00e084b 
> 
> Diff: https://reviews.apache.org/r/44958/diff/
> 
> 
> Testing
> ---
> 
> Manual testing:
> [root@c6401 sbin]# ps -ef | grep ambari-server
> root 20769 1 27 14:47 pts/000:00:24 
> //usr/jdk64/jdk1.8.0_60/bin/java -server -XX:NewRatio=3 
> -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit 
> -XX:CMSInitiatingOccupancyFraction=60 **-XX:+CMSPermGenSweepingEnabled 
> -XX:+CMSClassUnloadingEnabled** -Dsun.zip.disableMemoryMapping=true -Xms512m 
> -Xmx2048m **-XX:MaxPermSize=128m** 
> -Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
> -Djava.security.krb5.conf=/etc/krb5.conf 
> -Djavax.security.auth.useSubjectCredsOnly=false -Xms512m -Xmx2048m 
> -Djava.security.auth.login.config=//etc/ambari-server/conf/krb5JAASLogin.conf 
> -Djava.security.krb5.conf=/etc/krb5.conf 
> -Djavax.security.auth.useSubjectCredsOnly=false -cp 
> //etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/postgresql-jdbc.jar
>  org.apache.ambari.server.controller.AmbariServer
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Re: Review Request 44916: StackAdvisor needs to set config map for ams-grafana-env to meet non-null expectations



> On March 16, 2016, 1:57 p.m., Sid Wagle wrote:
> > Shouldn't we also have a non-null check in the backend code?

I am not sure.. I guess it depends on the expectations. Historically it appears 
that the property map is expected to not be null.  I am happy to add null 
protections in place I can find that need to be protected.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44916/#review123895
---


On March 16, 2016, 12:59 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44916/
> ---
> 
> (Updated March 16, 2016, 12:59 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Nate Cole, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15442
> https://issues.apache.org/jira/browse/AMBARI-15442
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The StackAdvisor needs to set a config map for the `ams-grafana-env`, even it 
> is empty, to meet non-null expectations in Kerberos and Blueprint logic. 
> 
> To to this, the following line needs to be added to the StackAdvisor code:
> 
> #stacks/HDP/2.0.6/services/stack_advisor.py:473
> ```
> putGrafanaProperty = self.putProperty(configurations, "ams-grafana-env", 
> services)
> ```
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 8c18421 
> 
> Diff: https://reviews.apache.org/r/44916/diff/
> 
> 
> Testing
> ---
> 
> Manually tested in cluster Kerbrerized using the Ambari UI.
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 44880: AMBARI-15436 Supportability: Create 'Reset UI' button in Ambari's experimental page

2016-03-19 Thread Yusaku Sako


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44880/#review123954
---




ambari-web/app/messages.js (line 3016)


Change to "Reset UI state"



ambari-web/app/messages.js (line 3017)


Proposed change:
"You should proceed only if the UI is misbehaving (such as incorrect 
navigation upon login, UI is stuck and won't loaded, etc).  Are you sure you 
want to reset the UI state?"


- Yusaku Sako


On March 16, 2016, 1:41 a.m., Zhe (Joe) Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44880/
> ---
> 
> (Updated March 16, 2016, 1:41 a.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly, Richard Zang, Srimanth Gunturi, Xi 
> Wang, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15436
> https://issues.apache.org/jira/browse/AMBARI-15436
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently Ambari UI persists it's current state on the server. This helps 
> when a different user logs in, they end up in the same UI. Sometimes however, 
> UI's state persisted on server does not sync with cluster state, thereby 
> putting UI in a broken state with no recourse for the stuck user.
> 
> Ambari UI should provide a _Reset UI_ button in the experimental page which 
> resets the UI state on server.
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/experimental.js 9fa6da5 
>   ambari-web/app/messages.js dc9c264 
>   ambari-web/app/templates/experimental.hbs 2873679 
> 
> Diff: https://reviews.apache.org/r/44880/diff/
> 
> 
> Testing
> ---
> 
> Local ambari-web test passed.
> 24578 tests complete (23 seconds)
> 145 tests pending
> Manual testing done.
> 
> 
> Thanks,
> 
> Zhe (Joe) Wang
> 
>

Re: Review Request 44835: AMBARI-15420: Refactor resource_management library


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123892
---


Ship it!





ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
 (line 106)


This may not necessarily be a rolling upgrade.



ambari-common/src/main/python/resource_management/libraries/script/script.py 
(line 383)


May want to ensure this is also a 3-tuple


- Alejandro Fernandez


On March 15, 2016, 8:23 p.m., Jayush Luniya wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> ---
> 
> (Updated March 15, 2016, 8:23 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo  
> Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Refactor resource_management library, to generalize the library and remove 
> HDP-specific hardcodings
> 
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of 
> hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven 
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of 
> hardcoding the "/usr/hdp"
> 
> In addition, also added a feature in the stack processing engine to support 
> properties values to be defined a external property file (See 
> tarball_map.json, stack_tools.json in patch)
> 
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), 
> get_tarball_map() are added in script.py, which will set the defaults if 
> these config properties are not defined (ambari-server upgrade scenario needs 
> to be addressed to add these config properties on upgrade). These helper 
> functions are used to remove hardcodings in resource_management library.
> 
> 
> Remaining HDP-specific logic in resource_management library
> 
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
> 
> Refactoring this HDP-specific logic would require "Stack Featurization" 
> (AMBARI-13364) to be in place.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py 
> ee7db0a 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  770595f 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  647b8b6 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/default.py
>  23383dc 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
>  a20b03c 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
>  f2e6567 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py
>  f1cd9cb 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py
>  d585dea 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py
>  4d9d8a4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
>  2ccc0c6 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  4a8eeb9 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py
>  95c5cba 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> a8098a0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  4be4049 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java
>  7f21aaa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> 0c7faea 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java
>  c739211 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> 7d934bb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java 
> e7c9c27 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java
>  3f7f

Review Request 44966: In IE, logging out of Ambari Admin View doesn't log user out of web console.

2016-03-19 Thread Sangeeta Ravindran


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44966/
---

Review request for Ambari, Alexandr Antonenko and Denys Buzhor.


Bugs: AMBARI-15319
https://issues.apache.org/jira/browse/AMBARI-15319


Repository: ambari


Description
---

Install latest Ambari trunk build.
Open Ambari web using IE. In the Admin view, click on Sign out.
The login page is displayed briefly and then the page returns to the Admin view.
This works fine on FF and Chrome.

This seems to be happening because of the additional parameters added to the 
url in signout() method in 
ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/Auth.js

// Workaround for sign off within Basic Authorization
  var origin = $window.location.protocol + '//' + Date.now() + ':' + 
Date.now() + '@' +
$window.location.hostname + ($window.location.port ? ':' + 
$window.location.port : '');
  return $http({
method: 'GET',
url: origin + Settings.baseUrl + '/logout'
  });

This causes an exception in IE (A security problem occurred.) when executing 
urlParsingNode.setAttribute("href", href); 

As a result, the server side performLogout method is not invoked and after 
temporarily displaying the login page, the user is brought back in to the 
Dashboard.

To fix this, we will instead pass an invalid username and password in the 
request header.

//use an invalid username and password in the request header
  $http.defaults.headers.common['Authorization'] = 'Basic ' + 
btoa('invalid_username:password');
  return $http({
method: 'GET',
url: Settings.baseUrl + '/logout'
   });


Diffs
-

  ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/Auth.js 
14c04c1 

Diff: https://reviews.apache.org/r/44966/diff/


Testing
---

Manually tested fix on IE, Chrome and FF.


Thanks,

Sangeeta Ravindran

Review Request 44983: Service version display should be based on Version Definition


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44983/
---

Review request for Ambari, Alejandro Fernandez and Jonathan Hurley.


Bugs: AMBARI-15473
https://issues.apache.org/jira/browse/AMBARI-15473


Repository: ambari


Description
---

Add {{stack_services}} to the repository_version endpoint, similar to the 
{{services}} attribute that lists the available services.

This will return a list of all stack services as they pertain to the Version 
Definition File.  This is required to show the correct versions that are 
available for a stack based on the repo_version.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RepositoryVersionResourceProvider.java
 c298e0a 
  
ambari-server/src/main/java/org/apache/ambari/server/state/repository/ManifestServiceInfo.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/state/repository/VersionDefinitionXml.java
 93ac767 
  
ambari-server/src/test/java/org/apache/ambari/server/state/repository/VersionDefinitionTest.java
 f2939c7 
  ambari-server/src/test/resources/version_definition_test.xml 69ea581 

Diff: https://reviews.apache.org/r/44983/diff/


Testing
---

Manual.  Automated pending.


Thanks,

Nate Cole

Re: Review Request 44773: Hive View : Upload Table : Tables are not deleted in case of exception.

2016-03-19 Thread Pallav Kulshreshtha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44773/#review123976
---


Ship it!




Ship It!

- Pallav Kulshreshtha


On March 14, 2016, 6:16 a.m., Nitiraj Rathore wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44773/
> ---
> 
> (Updated March 14, 2016, 6:16 a.m.)
> 
> 
> Review request for Ambari, DIPAYAN BHOWMICK, Pallav Kulshreshtha, and Yusaku 
> Sako.
> 
> 
> Bugs: AMBARI-15392
> https://issues.apache.org/jira/browse/AMBARI-15392
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Earlier : 
> In case of error the temporary table and actual table is was not getting 
> cleanedup (deleted).
> 
> In this patch:
> Now the rollback methods are added to UI which sends proper REST calls to 
> server for deletion of appropriate tables depending on the stage of flow 
> where error occurs.
> 
> 
> Diffs
> -
> 
>   
> contrib/views/hive/src/main/resources/ui/hive-web/app/controllers/upload-table.js
>  5085cc5 
> 
> Diff: https://reviews.apache.org/r/44773/diff/
> 
> 
> Testing
> ---
> 
> Manual testing done.
> 
> 
> Thanks,
> 
> Nitiraj Rathore
> 
>

Re: Review Request 45035: Restarting HDFS Before Upgrade Finalizing Does Not Supply the rollingUpgrade Flag


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45035/
---

(Updated March 18, 2016, 4:56 p.m.)


Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.


Changes
---

Using a marker file in addition to the upgrade state to figure out how to 
handle NN restart.


Bugs: AMBARI-15482
https://issues.apache.org/jira/browse/AMBARI-15482


Repository: ambari


Description
---

During an upgrade, the HDFS NameNode(s) are restarted with the 
{{-rollingUpgrade}} flag. However, it's possible to get to the end of an 
upgrade and decide to "Finalize Later".

This allows the cluster to run in the upgraded state before committing to the 
upgrade. Full cluster control is returned via the Ambari web client.

Administrators can then decide to restart a NameNode. Upon restarting the 
NameNode, it will produce an error that it was not started with the 
{{rollingUpgrade}} flag. 

It seems that as long as an upgrade has not been finalized, the NameNode(s) 
must be started with the {{rollingUpgrade}} to allow them to function properly. 

STR:
- Perform a rolling upgrade from HDP 2.2 to 2.3 (or 2.3 to 2.4); as long as 
there is a major version change.
- Before finalization, say "Finalize Later". All services should be up and 
green.
- Now restart a NameNode


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 9ea541e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
 4ef215c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 303f3a4 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 07061e1 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
 3f1a52b 
  ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
ed3c772 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 1c7ff61 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
 84bb9f3 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e01e693 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql fb24c89 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 2e092b2 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5f5df3c 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
3d71f1f 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 9d70fbb 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 715000e 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
 e4c8c9c 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
 02905ec 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py
 a154b73 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
 905802f 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
 17c52d2 
  ambari-web/app/controllers/main/admin/stack_and_upgrade_controller.js 2dceccc 
  ambari-web/app/utils/ajax/ajax.js 29d0715 

Diff: https://reviews.apache.org/r/45035/diff/


Testing
---

Pending...


Thanks,

Jonathan Hurley

Re: Review Request 44724: AMBARI-12906: Alert notifications are created even if credential fields are left empty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44724/#review124178
---




ambari-web/app/controllers/main/alerts/manage_alert_notifications_controller.js 
(line 516)


I'm not up-to-date on JavaScript syntax; is this also checking for empty 
values as well? Such as a bunch of spaces?


- Jonathan Hurley


On March 15, 2016, 3:17 p.m., Qin Liu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44724/
> ---
> 
> (Updated March 15, 2016, 3:17 p.m.)
> 
> 
> Review request for Ambari, Di Li and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-12906
> https://issues.apache.org/jira/browse/AMBARI-12906
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> User should not be allowed to save changes if the credential fields are left 
> empty when creating new Alert Notifications.
> 
> Steps to reproduce:
> 1. Create New Alert Notification
> 2. Enter Name = test
> 3. Tick the checkbox for 'Use Authentication'
> 4. Keep username and password empty
> 5. Click on Save
> 6. New alert notifications is created.
> 
> 
> Diffs
> -
> 
>   
> ambari-web/app/controllers/main/alerts/manage_alert_notifications_controller.js
>  79e524e 
>   ambari-web/app/messages.js 8e69dd0 
>   ambari-web/app/templates/main/alerts/create_alert_notification.hbs 0a83f63 
>   
> ambari-web/test/controllers/main/alerts/manage_alert_notifications_controller_test.js
>  f16bf4d 
> 
> Diff: https://reviews.apache.org/r/44724/diff/
> 
> 
> Testing
> ---
> 
> FIX:
> 1. With the fix, user won't be allowd to to save changes if Email is selected 
> for alert notification and Use authentication is chosen and the credential 
> fields are left empty when creating new Alert Notifications or editing 
> existing Alert Notifications.
> 2. The error message "Username is required" will be shown if the Username is 
> empty.
> 3. The error message "Password is required" will be shown if the password is 
> empty. 
> The fix here uses a similar approach that was used for the Password 
> Confirmation error handling.
> 
> TESTS:
> The fix has been tested with HDP UI.
> manage_alert_notifications_controller_test.js has also been modified to test 
> changes.
> 
> The following is the result of running ambari-web tests:
> 
> 24957 tests complete (31 seconds)
> 145 tests pending
> 
> 
> Thanks,
> 
> Qin Liu
> 
>

Re: Review Request 44916: StackAdvisor needs to set config map for ams-grafana-env to meet non-null expectations



> On March 16, 2016, 5:14 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessor.java,
> >  line 224
> > 
> >
> > How can this ever be null? An empty config type!?

This is not config properties from a service config type, but updates generated 
by the stack advisor. Up until now, this hasn't been null, but it is possible 
based on the data the stack advisor returns.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44916/#review123930
---


On March 16, 2016, 4:55 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44916/
> ---
> 
> (Updated March 16, 2016, 4:55 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Nate Cole, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15442
> https://issues.apache.org/jira/browse/AMBARI-15442
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The StackAdvisor needs to set a config map for the `ams-grafana-env`, even it 
> is empty, to meet non-null expectations in Kerberos and Blueprint logic. 
> 
> To to this, the following line needs to be added to the StackAdvisor code:
> 
> #stacks/HDP/2.0.6/services/stack_advisor.py:473
> ```
> putGrafanaProperty = self.putProperty(configurations, "ams-grafana-env", 
> services)
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessor.java
>  ea9fffb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  be6edc9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  f5e7578 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 8c18421 
>   
> ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessorTest.java
>  c4e694b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  f6027f3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  68d5755 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> 9070c0a 
> 
> Diff: https://reviews.apache.org/r/44916/diff/
> 
> 
> Testing
> ---
> 
> Manually tested in cluster Kerbrerized using the Ambari UI.
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 45012: Recommendation and validation logic for HAWQ hdfs-client.xml output.replace-datanode-on-failure property


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45012/
---

(Updated March 18, 2016, 7:47 p.m.)


Review request for Ambari, Alejandro Fernandez, bhuvnesh chaudhary, Goutam 
Tadi, jun aoki, Jayush Luniya, Lav Jain, and Sumit Mohanty.


Changes
---

updated magic number


Bugs: AMBARI-15499
https://issues.apache.org/jira/browse/AMBARI-15499


Repository: ambari


Description
---

Recommendation and validation logic for HAWQ hdfs-client.xml 
output.replace-datanode-on-failure property


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
 93ad3fe 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hdfs-client.xml
 c2a756a 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
c0bc4e2 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 4f0b4b7 

Diff: https://reviews.apache.org/r/45012/diff/


Testing
---

manual

--
Ran 253 tests in 7.111s

OK
--
Total run:927
Total errors:0
Total failures:0
OK


Thanks,

Alexander Denissov

Review Request 45035: Restarting HDFS Before Upgrade Finalizing Does Not Supply the rollingUpgrade Flag


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45035/
---

Review request for Ambari, Alejandro Fernandez, Nate Cole, and Robert Levas.


Bugs: AMBARI-15482
https://issues.apache.org/jira/browse/AMBARI-15482


Repository: ambari


Description
---

During an upgrade, the HDFS NameNode(s) are restarted with the 
{{-rollingUpgrade}} flag. However, it's possible to get to the end of an 
upgrade and decide to "Finalize Later".

This allows the cluster to run in the upgraded state before committing to the 
upgrade. Full cluster control is returned via the Ambari web client.

Administrators can then decide to restart a NameNode. Upon restarting the 
NameNode, it will produce an error that it was not started with the 
{{rollingUpgrade}} flag. 

It seems that as long as an upgrade has not been finalized, the NameNode(s) 
must be started with the {{rollingUpgrade}} to allow them to function properly. 

STR:
- Perform a rolling upgrade from HDP 2.2 to 2.3 (or 2.3 to 2.4); as long as 
there is a major version change.
- Before finalization, say "Finalize Later". All services should be up and 
green.
- Now restart a NameNode


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 9ea541e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
 4ef215c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 303f3a4 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 07061e1 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
 3f1a52b 
  ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
ed3c772 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 1c7ff61 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
 84bb9f3 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql a85202d 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 9b4810c 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql cc3d197 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 07c786d 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
ab6dc93 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 8e91fde 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 440ca44 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py
 e4c8c9c 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py
 02905ec 
  
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
 905802f 
  ambari-web/app/controllers/main/admin/stack_and_upgrade_controller.js 2dceccc 
  ambari-web/app/utils/ajax/ajax.js 29d0715 

Diff: https://reviews.apache.org/r/45035/diff/


Testing
---

Pending...


Thanks,

Jonathan Hurley

Re: Review Request 44916: StackAdvisor needs to set config map for ams-grafana-env to meet non-null expectations



> On March 17, 2016, 8:53 a.m., Jonathan Hurley wrote:
> > My only concern is for safety and clarify of code, does it makes sense to 
> > change the contract of the config to always return a non-null (if empty) 
> > map?

I attempted to go this route, but I got stuck where the Jackson JSON parser 
comes into play and fills the 
`org.apache.ambari.server.api.services.stackadvisor.recommendations.RecommendationResponse.BlueprintConfigurations`
 object.  This is where the null is getting set - or rather not set.  So if we 
wanted to persue this route, I can set a default value for the relevant class 
member and force it to never be null.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44916/#review124010
---


On March 16, 2016, 4:55 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44916/
> ---
> 
> (Updated March 16, 2016, 4:55 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Nate Cole, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15442
> https://issues.apache.org/jira/browse/AMBARI-15442
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The StackAdvisor needs to set a config map for the `ams-grafana-env`, even it 
> is empty, to meet non-null expectations in Kerberos and Blueprint logic. 
> 
> To to this, the following line needs to be added to the StackAdvisor code:
> 
> #stacks/HDP/2.0.6/services/stack_advisor.py:473
> ```
> putGrafanaProperty = self.putProperty(configurations, "ams-grafana-env", 
> services)
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessor.java
>  ea9fffb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  be6edc9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  f5e7578 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 8c18421 
>   
> ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessorTest.java
>  c4e694b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  f6027f3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  68d5755 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> 9070c0a 
> 
> Diff: https://reviews.apache.org/r/44916/diff/
> 
> 
> Testing
> ---
> 
> Manually tested in cluster Kerbrerized using the Ambari UI.
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Review Request 44965: AMBARI-15452 : Change timeouts for Hbase and Phoenix

2016-03-19 Thread Aravindan Vijayan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44965/
---

Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-15452
https://issues.apache.org/jira/browse/AMBARI-15452


Repository: ambari


Description
---

Change a few timeouts in AMS Hbase and Phoenix to 5 minutes in Hbase-site.xml

hbase.client.scanner.timeout.period = 30?
hbase.rpc.timeout = 30
phoenix.query.timeoutMs = 30
phoenix.query.keepAliveMs = 30


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
 8267d5d 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-hbase-site.xml
 acd14aa 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java
 82d5464 

Diff: https://reviews.apache.org/r/44965/diff/


Testing
---

Unit tests pass locally.


Thanks,

Aravindan Vijayan

Re: Review Request 44688: PXF should reach out to active namenode to get the delegation token on HDFS HA clusters


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44688/#review123919
---


Ship it!




Ship It!

- Jayush Luniya


On March 11, 2016, 1:26 a.m., bhuvnesh chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44688/
> ---
> 
> (Updated March 11, 2016, 1:26 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, 
> Matt, and Oleksandr Diachenko.
> 
> 
> Bugs: AMBARI-15380
> https://issues.apache.org/jira/browse/AMBARI-15380
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> PXF alerts are not working on secured HDFS HA clusters. When the cluster is 
> HA, Active Namenode should be used to get the delegation token for pxf, 
> however currently it goes to localhost.
> Updated the logic to find out the active namenode and use it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/package/alerts/api_status.py
>  871da2b 
> 
> Diff: https://reviews.apache.org/r/44688/diff/
> 
> 
> Testing
> ---
> 
> yes. manually.
> 
> 
> Thanks,
> 
> bhuvnesh chaudhary
> 
>

Re: Review Request 44688: PXF should reach out to active namenode to get the delegation token on HDFS HA clusters



> On March 16, 2016, 8 p.m., Jayush Luniya wrote:
> > Ship It!

Thank you Jayush. No worries, will do merge now.


- bhuvnesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44688/#review123919
---


On March 11, 2016, 1:26 a.m., bhuvnesh chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44688/
> ---
> 
> (Updated March 11, 2016, 1:26 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, 
> Matt, and Oleksandr Diachenko.
> 
> 
> Bugs: AMBARI-15380
> https://issues.apache.org/jira/browse/AMBARI-15380
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> PXF alerts are not working on secured HDFS HA clusters. When the cluster is 
> HA, Active Namenode should be used to get the delegation token for pxf, 
> however currently it goes to localhost.
> Updated the logic to find out the active namenode and use it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/package/alerts/api_status.py
>  871da2b 
> 
> Diff: https://reviews.apache.org/r/44688/diff/
> 
> 
> Testing
> ---
> 
> yes. manually.
> 
> 
> Thanks,
> 
> bhuvnesh chaudhary
> 
>

Re: Review Request 45056: Blueprint install using config_recommendation_strategy is not functional

2016-03-19 Thread Oliver Szabo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45056/#review124404
---


Ship it!




Ship It!

In the near future I will create a patch for fixing some issues with stack 
advisor blueprint support (e.g.: same kind of error can happen if hosts are not 
registered during stack advisor processing)

- Oliver Szabo


On March 18, 2016, 10:31 p.m., Shantanu Mundkur wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45056/
> ---
> 
> (Updated March 18, 2016, 10:31 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo and Robert Levas.
> 
> 
> Bugs: AMBARI-15454
> https://issues.apache.org/jira/browse/AMBARI-15454
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> hBlueprint install using config_recommendation strategy seems to hang for a 
> long time (couple of hours?) and ends up logging exceptions continually to 
> ambari-server.log. At the same time many hundreds of directories are seen 
> getting created under /var/run/ambari-server/stack-recommendations (I have 
> seen above 800-900). If you keep it running eventually the cluster install 
> seems to start but fails miserably at least during the start and some of it 
> makes obvious that configuration recommendations were NOT applied. You see 
> errors during startup that hint that options used for JVM (For Datanode etc) 
> were unreasonable.
> 
> Note that both the blueprint and cluster templates used empty configurations. 
> Example:
> 
> .
> .
>"configurations" : [],
>"host_groups": [
> {
>  "name": "host-group-1",
>  "configurations" : [],
>  "cardinality" : "1",
>  "components": [
>   { "name": "APP_TIMELINE_SERVER" },
>   { "name": "DATANODE" },
>   { "name": "FALCON_CLIENT" },
>   { "name": "FALCON_SERVER" },
>   { "name": "FLUME_HANDLER" },
>   { "name": "HBASE_CLIENT" },
>   { "name": "HBASE_MASTER" },
>   { "name": "HBASE_REGIONSERVER" },
> .
> .
> .
> 
> Teh cluster template was:
> { 
> "blueprint": "1node",
> "config_recommendation_strategy" : "ONLY_STACK_DEFAULTS_APPLY",
> "default_password": "myPassword1",
> "host_groups": [
> {
> "name": "host-group-1",
> "hosts": [
> {
> "fqdn": "mynode.ibm.com"
> }
> ]
> }
> ] 
> }
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  f5e7578 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  68d5755 
> 
> Diff: https://reviews.apache.org/r/45056/diff/
> 
> 
> Testing
> ---
> 
> 1) ambari-server unit tests
> 
> Results :
> 
> Tests run: 3968, Failures: 0, Errors: 0, Skipped: 33
> 2) Added unit test to 
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>
> 3) Deployed clusters 1-5 nodes specifying "config_recommendation_strategy" : 
> "ONLY_STACK_DEFAULTS_APPLY"
> 
> 
> Thanks,
> 
> Shantanu Mundkur
> 
>

Review Request 44915: https://issues.apache.org/jira/browse/HAWQ-497

2016-03-19 Thread Goutam Tadi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44915/
---

Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, jun aoki, 
and Lav Jain.


Bugs: AMBARI-15435
https://issues.apache.org/jira/browse/AMBARI-15435


Repository: ambari


Description
---

Update configs for setting virtual segments in HAWQ HashDistributed Tables and 
random distributed tables


Diffs
-

  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
 d3aa0ed 
  ambari-server/src/main/resources/common-services/HAWQ/2.0.0/themes/theme.json 
aa35311 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
791d5b8 
  ambari-web/app/messages.js 5addc9b 

Diff: https://reviews.apache.org/r/44915/diff/


Testing
---

Yes. Manually.


Thanks,

Goutam Tadi

Re: Review Request 44880: AMBARI-15436 Supportability: Create 'Reset UI' button in Ambari's experimental page

2016-03-19 Thread Zhe (Joe) Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44880/
---

(Updated March 17, 2016, 1:13 a.m.)


Review request for Ambari, Jaimin Jetly, Richard Zang, Srimanth Gunturi, Xi 
Wang, and Yusaku Sako.


Bugs: AMBARI-15436
https://issues.apache.org/jira/browse/AMBARI-15436


Repository: ambari


Description
---

Currently Ambari UI persists it's current state on the server. This helps when 
a different user logs in, they end up in the same UI. Sometimes however, UI's 
state persisted on server does not sync with cluster state, thereby putting UI 
in a broken state with no recourse for the stuck user.

Ambari UI should provide a _Reset UI_ button in the experimental page which 
resets the UI state on server.


Diffs (updated)
-

  ambari-web/app/controllers/experimental.js 9fa6da5 
  ambari-web/app/mappers/server_data_mapper.js 1e08161 
  ambari-web/app/messages.js 9289fc2 
  ambari-web/app/templates/experimental.hbs 2873679 

Diff: https://reviews.apache.org/r/44880/diff/


Testing
---

Local ambari-web test passed.
24578 tests complete (23 seconds)
145 tests pending
Manual testing done.


Thanks,

Zhe (Joe) Wang

Re: Review Request 44926: [DRAFT] Auto-retry on failure during RU/EU


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44926/#review123927
---




ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (lines 255 - 262)


Can be only one, even if it's a downgrade.



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 277)


I think you know why this and others like it can't go into a commit :)


- Nate Cole


On March 16, 2016, 4:41 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44926/
> ---
> 
> (Updated March 16, 2016, 4:41 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-15446
> https://issues.apache.org/jira/browse/AMBARI-15446
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a failure occurs during RU/EU and the task transitions to HOLDING_FAILED 
> or HOLDING_TIMEDOUT, want Ambari to automatically retry up to up to x mins. 
> This is useful when a host goes down as Ambari is running a task on it.
> ambari.properties will have 1 new parameter. E.g,. 
> stack-upgrade.max_retry_timeout_mins=15 (by default, will not be present)
> If Ambari Server is restarted, it should be able to recover.
> Today, Action Scheduler increases the attempt_count whenever a task is 
> retried, but it requires resetting the start_time to -1. Because of this, we 
> cannot rely on the start_time property to know when to timeout after several 
> retries.
> 
> For the implementation, will add another thread to Ambari that will monitor 
> failed tasks only during active RU/EU and change the status back to PENDING 
> so that Action Scheduler can reschedule it.
> Luckily, any tasks in HOLDING_TIMEDOUT and HOLDING_FAILED states are 
> blocking, so no other stages are allowed to proceed.
> In order to know when a task was first started, will add a new property to 
> host_role_command table called original_start_time.
> 
> For the agents, we need to ensure that they always write out a response. On 
> the first heartbeat, it should send the status of its last command so we know 
> it failed and Ambari can retry.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  3a80803 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/PreviousUpgradeCompleted.java
>  3a4467f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/ClusterVersionDAO.java
>  1bcca60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
>  f5b1cb4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/ClusterVersionEntity.java
>  f1867b4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java
>  19f0602 
>   ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 
> ed3c772 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  1c7ff61 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/LogicalRequest.java
>  82edbcf 
> 
> Diff: https://reviews.apache.org/r/44926/diff/
> 
> 
> Testing
> ---
> 
> Verified on a live cluster.
> 
> TODO: Still need to make more changes to the implementation, add the config, 
> switch to gauva service, add a column, and add unit tests.
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 44933: Apply the stack featurization prototype detailed on AMBARI-13364 to ZOOKEEPER service.


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44933/#review124285
---




ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py
 (line 1)


Minor, fix typo in file name



ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
 (line 53)


Will things might work out in regular install, I think there is going to be 
a problem here during upgrades.

Lets say we are doing EU from HDP-2.1 to HDP-2.2
the command.json file for the command will have 
hostLevelParams/current_version = 2.1.0.0-1234
hostLevelParams/version = 2.2.0.0-5678
commandParams/stack_version = 2.2
configurations/cluster-env/stack_features = "snappy" (HDP-2.1 version of 
stack_features)

So the check will fail.

@Alejandro is my understanding correct? I can try out the upgrade to verify 
too. 

BTW, @Juanjo Script.is_stack_greater* functions use 
commandParams/stack_version whereas you are changing it to current_version



ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml 
(line 25)


FYI, in https://reviews.apache.org/r/44835/ I added a feature to support 
external properties file for complete structures like these. So you can create 
a json file at HDP/2.0.6/properties/stack_features.json and reference it in the 
config property xml file. I will be committing those changes soon.



ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 
(lines 29 - 32)


Dont need to provide min_version if the feature was not enabled at from a 
maint release onwards.



ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml 
(lines 29 - 35)


Dont need to provide min_version if the feature was not enabled at from a 
maint release onwards.


- Jayush Luniya


On March 16, 2016, 9:24 p.m., Juanjo  Marron wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44933/
> ---
> 
> (Updated March 16, 2016, 9:24 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Jayush Luniya.
> 
> 
> Bugs: AMBARI-14435
> https://issues.apache.org/jira/browse/AMBARI-14435
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Apply the  stack featurization prototype  detailed on AMBARI-13364 to 
> ZOOKEEPER service.
> This patch removes the hardcoded stack versions in ZK common-services code 
> (centralized version and parameterized stack configurations will be faced in 
> other JIRAS)
> stack_features is now a property in .json format in 
> configuration/cluster-env.xml for each stack version.
> With the structure:
> {
> "stack_features": [
> { "name": "feature1", "description" : "Feature1 support", "min_version" : 
> "Y.Y.Y.Y" , "max_version" : "X.X.X.X"}
> , 
> ...
> ]
> }
> where min_version/max_version are optional constraints.
> New resource_management/libraries/functions/stack_fetaures.py has been 
> introduced to parse the json file and called from service code to check if 
> the stack supports the required feature.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_fetaures.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
>  9ff9125 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/status_params.py
>  d18e4d7 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper.py
>  5712ce4 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_client.py
>  25ace24 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py
>  d5b6898 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_service.py
>  6fe0772 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml
>  3fb82e9 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/configuration/cluster-env.xml 
> f1fa4de 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/configuration/cluster-env.xml 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/44933/diff/
> 
> 
> Testing
> ---
> 
> Zookeeper fresh installation has been tested using HDP stack 2.3 and 
> AMBARI-13364 branch code.
> 
> 
> Thanks,
> 
>

Review Request 44955: Yarn start is failed (YARN Client Install) after upgrade from 1.7.0 to 2.2.2.0


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44955/
---

Review request for Ambari and Vitalyi Brodetskyi.


Bugs: AMBARI-15461
https://issues.apache.org/jira/browse/AMBARI-15461


Repository: ambari


Description
---

STR: *  
1)Deploy old version  
2)Enable/Disable MIT security  
3)Make Ambari only Upgrade

**Actual result:**  
Yarn start is failed (YARN Client Install) after upgrade from 1.7.0 to 2.2.2.0




stderr:   /var/lib/ambari-agent/data/errors-1125.txt

Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py",
 line 65, in 
YarnClient().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 219, in execute
method(env)
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py",
 line 34, in install
self.configure(env)
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py",
 line 37, in configure
import params
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/params.py",
 line 28, in 
from params_linux import *
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py",
 line 142, in 
rm_https_port = 
config['configurations']['yarn-site']['yarn.resourcemanager.webapp.https.address'].split(':')[-1]
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
 line 81, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found 
in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 
'yarn.resourcemanager.webapp.https.address' was not found in configurations 
dictionary!


Diffs
-

  
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
 c3a499e 

Diff: https://reviews.apache.org/r/44955/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 45030: Replace 0.0.0.0 with hostname in ams-site/timeline.metrics.service.webapp.address


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45030/#review124204
---


Ship it!




Ship It!

- Jonathan Hurley


On March 18, 2016, 11:33 a.m., Dmytro Sen wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45030/
> ---
> 
> (Updated March 18, 2016, 11:33 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Robert Nettleton, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15480
> https://issues.apache.org/jira/browse/AMBARI-15480
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> We need collector hostname in configs. 
> Replace 0.0.0.0 with hostname in 
> ams-site/timeline.metrics.service.webapp.address
> Implement changes for blueprints, stack_advisor, upgrade catalog
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  f5e7578 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
>  84bb9f3 
>   
> ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/params.py
>  a21035e 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-site.xml
>  99e97f7 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
>  12c0f25 
>   
> ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/params.py
>  29f71a7 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
>  b32aedf 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/params.py
>  d3a0606 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1.2.1/package/scripts/params_linux.py
>  ab935a1 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
>  7153c54 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> ef82bce 
>   
> ambari-server/src/main/resources/stacks/HDPWIN/2.1/hooks/before-START/scripts/params.py
>  469e709 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  68d5755 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog222Test.java
>  acbf055 
>   ambari-server/src/test/python/stacks/2.0.6/configs/default.json b23d4b9 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_ams_embedded.json 
> 688dbe0 
>   
> ambari-server/src/test/python/stacks/2.0.6/configs/default_hive_non_hdfs.json 
> 6d92182 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> 921ee23 
>   
> ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade-hdfs-secure.json
>  a43f15a 
>   ambari-server/src/test/python/stacks/2.2/configs/journalnode-upgrade.json 
> 59cd868 
>   ambari-server/src/test/python/stacks/2.2/configs/ranger-admin-upgrade.json 
> 481b6d4 
>   
> ambari-server/src/test/python/stacks/2.2/configs/ranger-usersync-upgrade.json 
> 200ad23 
>   ambari-server/src/test/python/stacks/2.3/configs/ats_1_5.json 3fe1235 
> 
> Diff: https://reviews.apache.org/r/45030/diff/
> 
> 
> Testing
> ---
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 44859: Cannot specify ZooKeeper heap size via UI

2016-03-19 Thread Dmytro Sen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44859/#review124013
---




ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
 (line 68)


default 1024m instead of 1024 ?


- Dmytro Sen


On Март 17, 2016, 1:22 п.п., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44859/
> ---
> 
> (Updated Март 17, 2016, 1:22 п.п.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Dmytro 
> Sen.
> 
> 
> Bugs: AMBARI-15429
> https://issues.apache.org/jira/browse/AMBARI-15429
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> zookeeper-env template has the following:
> export SERVER_JVMFLAGS=zk_server_heapsize
> However, we don't have any heap size property that is exposed via the UI, so 
> 1GB heapsize is always used.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/configuration/zookeeper-env.xml
>  7ef62bc 
>   
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/params_linux.py
>  9ff9125 
> 
> Diff: https://reviews.apache.org/r/44859/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 44886: AMBARI-15437 : Handle changing symlinks of ranger admin binaries for upgrade scenarios

2016-03-19 Thread Gautam Borad


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44886/
---

(Updated March 17, 2016, 10:13 a.m.)


Review request for Ambari, Alejandro Fernandez, Jaimin Jetly, Jonathan Hurley, 
Mahadev Konar, Sumit Mohanty, Selvamohan Neethiraj, and Velmurugan Periasamy.


Changes
---

Using 2.3 stack or greater flag and checking link exist condition


Bugs: AMBARI-15437
https://issues.apache.org/jira/browse/AMBARI-15437


Repository: ambari


Description
---

For Ambari stacks for Ranger: 
We need to take care of handling symlinks to point to current version of HDP, 
to avoid confusion.


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
 6394f51 

Diff: https://reviews.apache.org/r/44886/diff/


Testing
---

Tested EU and RU with Ranger installed.
Testing steps:
1) Installed Ranger with HDP-2.2.x
2) Upgrade Ranger to HDP-2.3.x
3) Again Upgrade to HDP-2.4.x

Ran 277 tests in 7.293s

OK
--
Total run:941
Total errors:0
Total failures:0
OK


Thanks,

Gautam Borad

Re: Review Request 44956: Topology host info is not cleared when a host is removed

2016-03-19 Thread Sebastian Toader


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44956/#review124030
---




ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
 (line 780)


Please ensure that this method runs within a transaction by using 
@Transactional annotation



ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopology.java
 (line 175)


Looks like unfinished comment


- Sebastian Toader


On March 17, 2016, 1:59 p.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44956/
> ---
> 
> (Updated March 17, 2016, 1:59 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Laszlo Puskas, Oliver Szabo, 
> Sandor Magyari, Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-15457
> https://issues.apache.org/jira/browse/AMBARI-15457
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a host is removed, a record is left in the topology_host_info table. As 
> a result re-adding the same host is not possible, due to invalid topology.
> Host removal should remove the corresponding record from topology_host_info 
> as well.
> 
> A new column with a foreign key to hosts table is added. In addition 
> ClusterTopologyImpl was statefull and hostGroupInfoMap was not updated when a 
> host was removed.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
>  6101ee8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/TopologyHostInfoDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/TopologyHostInfoEntity.java
>  a4f251a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
>  46dd49d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopology.java
>  4e178c0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java
>  003539c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupInfo.java
>  4847a11 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedState.java
>  dbf6735 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedStateImpl.java
>  b878955 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java
>  41f3150 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
>  8267d5d 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql dee397f 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 253b5d7 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 832d1bc 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> 6c8a09d 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a6acb74 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 2957c5c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  3e43ef1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostResourceProviderTest.java
>  69c570e 
> 
> Diff: https://reviews.apache.org/r/44956/diff/
> 
> 
> Testing
> ---
> 
> Tests ran without failures. (2016-03-17 13:30)
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Re: Review Request 44725: After exporting blueprint from ranger enabled cluster ranger.service.https.attrib.keystore.pass is exported

2016-03-19 Thread Amruta Borkar



> On March 14, 2016, 12:27 p.m., Di Li wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java,
> >  line 2668
> > 
> >
> > So any properties end with ".pass" will be marked, yes? That check 
> > seems a bit broad to me, at least broader than checking Password or Secret.
> > 
> > Each property (understandably that only when its defintion follows the 
> > rules) should use "property-type" to indicate if it's a password 
> > "PASSWORD". Can we check this instead of 
> > guessing property name patterns? 
> > 
> > We should honor the setting, so that when user needs to mask a 
> > property, he can either ends the property name with .password or .secret 
> > (this is really a hardcoded logic than stack driven) or configure the 
> > property propertly (stack driven, a more preferrable way).

Hello Di,

Property_type is not accessible to the DAO part. It shows only at UI. So I have 
tried to generalise the expression using Property_value rather than 
property_name. Please give your input for that. 
Thanks


- Amruta


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44725/#review123409
---


On March 17, 2016, 6:52 p.m., Amruta Borkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44725/
> ---
> 
> (Updated March 17, 2016, 6:52 p.m.)
> 
> 
> Review request for Ambari, Di Li and Robert Nettleton.
> 
> 
> Bugs: AMBARI-15338
> https://issues.apache.org/jira/browse/AMBARI-15338
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After exporting blueprint from ranger enabled cluster 
> ranger.service.https.attrib.keystore.pass is exported.
> Which needs to be removed before using the same blueprint to create another 
> cluster
> Error Show when used same blueprint:
> { "status" : 400, "message" : "Blueprint configuration validation failed: 
> Secret references are not allowed in blueprints, replace following properties 
> with real passwords:\n Config:ranger-admin-site 
> Property:ranger.service.https.attrib.keystore.pass\n" }
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  f5e7578 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  68d5755 
> 
> Diff: https://reviews.apache.org/r/44725/diff/
> 
> 
> Testing
> ---
> 
> Modified test cases to test for if the properties that end with "pass" are 
> getting filtered. Other properties which have 'pass' else where in the name 
> will not get filtered.
> 
> 
> Thanks,
> 
> Amruta Borkar
> 
>

Re: Review Request 44835: AMBARI-15420: Refactor resource_management library



> On March 16, 2016, 5:53 p.m., Alejandro Fernandez wrote:
> > ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py,
> >  line 106
> > 
> >
> > This may not necessarily be a rolling upgrade.

fixed


> On March 16, 2016, 5:53 p.m., Alejandro Fernandez wrote:
> > ambari-common/src/main/python/resource_management/libraries/script/script.py,
> >  line 384
> > 
> >
> > May want to ensure this is also a 3-tuple

padded the result to always return 3-element tuple


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/#review123892
---


On March 18, 2016, 7:26 p.m., Jayush Luniya wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44835/
> ---
> 
> (Updated March 18, 2016, 7:26 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo  
> Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-15420
> https://issues.apache.org/jira/browse/AMBARI-15420
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Refactor resource_management library, to generalize the library and remove 
> HDP-specific hardcodings
> 
> Specifically,
> 1. Make stack-selector, conf-selector tools stack-driven instead of 
> hardcoding hdp-select and conf-select
> 2. Make stack-root stack-driven instead of hardcoding /usr/hdp
> 3. Make copy_tarball mappings stack-driven 
> 4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of 
> hardcoding the "/usr/hdp"
> 
> In addition, also added a feature in the stack processing engine to support 
> properties values to be defined a external property file (See 
> tarball_map.json, stack_tools.json in patch)
> 
> Three config properties are added
> 1. cluster-env/stack_root
> 2. cluster-env/stack_tools
> 3. cluster-env/tarball_map
> Corresponding helper functions get_stack_root(), get_stack_tool(), 
> get_tarball_map() are added in script.py, which will set the defaults if 
> these config properties are not defined (ambari-server upgrade scenario needs 
> to be addressed to add these config properties on upgrade). These helper 
> functions are used to remove hardcodings in resource_management library.
> 
> 
> Remaining HDP-specific logic in resource_management library
> 
> 1. conf_select::_valid()
> 2. conf_select::get_hadoop_conf_dir()
> 3. list_ambari_managed_repos::repository_names
> 4. version_select_util::get_component_version()
> 5. script::get_stack_version()
> 6. script::should_expose_component_version()
> 7. get_lzo_packages::get_lzo_packages()
> 
> Refactoring this HDP-specific logic would require "Stack Featurization" 
> (AMBARI-13364) to be in place.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py 
> ee7db0a 
>   ambari-common/src/main/python/resource_management/core/utils.py 71d0008 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
>  770595f 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
>  647b8b6 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/default.py
>  23383dc 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
>  a20b03c 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
>  f2e6567 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py
>  f1cd9cb 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py
>  d585dea 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py
>  4d9d8a4 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
>  2ccc0c6 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
>  4a8eeb9 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py
>  PRE-CREATION 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py
>  95c5cba 
>   
> ambari-common/src/main/python/resource_management/libraries/script/script.py 
> 1b0116e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  4be4049 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java
>  7f21aaa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/Ser

Re: Review Request 44962: file_system get_mount_point_for_dir works incorrect


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44962/#review124066
---


Ship it!




Ship It!

- Alejandro Fernandez


On March 17, 2016, 4:03 p.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44962/
> ---
> 
> (Updated March 17, 2016, 4:03 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez and Dmytro Sen.
> 
> 
> Bugs: AMBARI-15468
> https://issues.apache.org/jira/browse/AMBARI-15468
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> **Case 1**:
> 
> 
> [root@c6403 ~]# mount
> /dev/mapper/VolGroup-lv_root on / type ext4 (rw)
> /newFs on /newHdfsDir type ext3 (rw,loop=/dev/loop1)
> 
> [root@c6403 ~]# python
> >>> from resource_management import *
> >>> from resource_management.libraries.functions import file_system
> >>> with Environment() as env:
> ...   print file_system.get_mount_point_for_dir('/newHdfsDir/some_dir')
> ...   print file_system.get_mount_point_for_dir('/newHdfsDir')
> ... 
> 2016-03-17 13:00:33,640 - Mount point for directory /newHdfsDir/some_dir 
> is /
> 2016-03-17 13:00:33,640 - Mount point for directory /newHdfsDir is /
> 
> 
> It should be /newHdfsDir, not /
> 
> **Reason**   
> /newHdfsDir and / have the same count of os separators.
> 
> **Case 2**:
> 
> 
> [root@c6403 ~]# mount
> /dev/mapper/VolGroup-lv_root on / type ext4 (rw)
> /newFs on /a/a type ext3 (rw,loop=/dev/loop2)
> /newFs on /a/a1 type ext3 (rw,loop=/dev/loop3)
> 
> [root@c6403 ~]# python
> >>> from resource_management import *
> >>> from resource_management.libraries.functions import file_system
> >>> with Environment() as env:
> >>> from resource_management import *
> >>> from resource_management.libraries.functions import file_system
> >>> with Environment() as env:
> ...   print file_system.get_mount_point_for_dir('/a/a1/some_dir')
> ...   print file_system.get_mount_point_for_dir('/a/a1')
> ...
> 2016-03-17 13:07:38,899 - Mount point for directory /a/a1/some_dir is /a/a
> 2016-03-17 13:07:38,900 - Mount point for directory /a/a1 is /a/a
> 
> 
> It should be /a/a1, not /a/a
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/test/python/resource_management/TestFileSystem.py 4e0eb63 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/file_system.py
>  39b86dd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 6132c2c 
> 
> Diff: https://reviews.apache.org/r/44962/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Andrew Onischuk
> 
>

Re: Review Request 44956: Topology host info is not cleared when a host is removed

2016-03-19 Thread Oliver Szabo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44956/#review124240
---


Ship it!




Ship It!

- Oliver Szabo


On March 18, 2016, 9:27 a.m., Daniel Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44956/
> ---
> 
> (Updated March 18, 2016, 9:27 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Laszlo Puskas, Oliver Szabo, 
> Sandor Magyari, Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-15457
> https://issues.apache.org/jira/browse/AMBARI-15457
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a host is removed, a record is left in the topology_host_info table. As 
> a result re-adding the same host is not possible, due to invalid topology.
> Host removal should remove the corresponding record from topology_host_info 
> as well.
> 
> A new column with a foreign key to hosts table is added. In addition 
> ClusterTopologyImpl was statefull and hostGroupInfoMap was not updated when a 
> host was removed.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java
>  6101ee8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/TopologyHostInfoDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/TopologyHostInfoEntity.java
>  a4f251a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
>  46dd49d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopology.java
>  4e178c0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java
>  003539c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupInfo.java
>  4847a11 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedState.java
>  dbf6735 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedStateImpl.java
>  b878955 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java
>  41f3150 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog222.java
>  8267d5d 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql dee397f 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 253b5d7 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 832d1bc 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> 6c8a09d 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql a6acb74 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 2957c5c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  3e43ef1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostResourceProviderTest.java
>  69c570e 
> 
> Diff: https://reviews.apache.org/r/44956/diff/
> 
> 
> Testing
> ---
> 
> Tests ran without failures. (2016-03-17 13:30)
> 
> 
> Thanks,
> 
> Daniel Gergely
> 
>

Re: Review Request 44688: PXF should reach out to active namenode to get the delegation token on HDFS HA clusters



> On March 14, 2016, 5:10 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/PXF/3.0.0/package/alerts/api_status.py,
> >  line 140
> > 
> >
> > Check for null (i.e. no active namenode)
> 
> bhuvnesh chaudhary wrote:
> Thanks for review Jayush,
> 
> I was testing the behavior when the Namenode is down to essentially test 
> for no active namenode, and 
> https://github.com/apache/ambari/blob/trunk/ambari-common/src/main/python/resource_management/libraries/functions/namenode_ha_utils.py#L135-L143
>  handles if the active_namenode identified is an empty list, and returns 
> UnknownConfiguration(..) if none is found which handles the null / empty case 
> ? So, thought not to handle it here again.
> 
> On the other hand, the response from get_active_namenode returned is 
> 'Configuration parameter 'fs_root' was not found in configurations 
> dictionary!' when no active namenode is found (i turned down HDFS), which is 
> kind of not explanatory, do you think we should make this as an exception in 
> a separate jira ?
> 
> bhuvnesh chaudhary wrote:
> Jayush, Let me know if there is any concern if i drop the issue based on 
> the response provided up.

Dropping the issue and will merge it. Will revisit in case needed.


- bhuvnesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44688/#review123456
---


On March 11, 2016, 1:26 a.m., bhuvnesh chaudhary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44688/
> ---
> 
> (Updated March 11, 2016, 1:26 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, jun aoki, Jayush Luniya, 
> Matt, and Oleksandr Diachenko.
> 
> 
> Bugs: AMBARI-15380
> https://issues.apache.org/jira/browse/AMBARI-15380
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> PXF alerts are not working on secured HDFS HA clusters. When the cluster is 
> HA, Active Namenode should be used to get the delegation token for pxf, 
> however currently it goes to localhost.
> Updated the logic to find out the active namenode and use it.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/package/alerts/api_status.py
>  871da2b 
> 
> Diff: https://reviews.apache.org/r/44688/diff/
> 
> 
> Testing
> ---
> 
> yes. manually.
> 
> 
> Thanks,
> 
> bhuvnesh chaudhary
> 
>

Re: Review Request 44986: AMBARI-15474: Listen for changes to auto-start configuration and send them to the agent during heartbeats

2016-03-19 Thread Nahappan Somasundaram


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44986/#review124119
---




ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentRecoveryChangeEvent.java
 (line 1)


Rename to ServiceComponentChangedEvent to be generic. It can be used 
whenever a property changes.


- Nahappan Somasundaram


On March 17, 2016, 4 p.m., Nahappan Somasundaram wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44986/
> ---
> 
> (Updated March 17, 2016, 4 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15474
> https://issues.apache.org/jira/browse/AMBARI-15474
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-15474: Listen for changes to auto-start configuration and send them to 
> the agent during heartbeats.
> 
> In HeartbeatHandler::handleHeartBeat(), send the recovery configuration to 
> the agent. Instead of pulling the recovery configuration from the application 
> cache, implement change events and get notified whenever there are changes to 
> the recovery configuration.
> 
> **Changes**:
> 1. Instead of sending the recovery configuration to the agent during every 
> heartbeat (10 seconds interval), send the configuration only when changes are 
> detected.
> 2. Used AmbariEventPublisher to publish the changes. RecoveryConfigHelper is 
> the subscriber which listens to changes to the configuration.
> 3. RecoveryConfigHelper maintains a map of Cluster-->Host-->Timestamp which 
> is the last updated timestamp of the recovery configuration. This timestamp 
> is used as a token and is sent to the agent with the configuration. The agent 
> provides this timestamp during each heartbeat. If the timestamp from the 
> heartbeat is the same as the one on the server side, no configuration is sent 
> to the agent.
> 4. Whenever changes are detected, the timestamp is invalidated. During the 
> next heartbeat, the recovery configuration is read from the DB and sent to 
> the agent along with the latest timestamp.
> 5. Added new unit tests to test each configuration change event.
> 
> 
> Diffs
> -
> 
>   ambari-agent/src/main/python/ambari_agent/Controller.py 
> c1c16ac97cca0d2677da2e26eee7b4949a4bf15c 
>   ambari-agent/src/main/python/ambari_agent/Heartbeat.py 
> b28644c834a2387d2fb0ad17d104224e830a0245 
>   ambari-agent/src/main/python/ambari_agent/RecoveryManager.py 
> ed537ca6928ce376c5f3e906f7a47c3f1919e309 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeat.java 
> 1430aa20cf68db071be85c4ad2ebb9acdaccecc0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  3a80803724bcb6ca9e4ec875aa17c9ac1c66fbe8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
>  617b04b5b8768ba28a3c8ecb6e5e00601f153396 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfig.java
>  c2c18462c586292caaf40bdb6a25a8fe9a39d76c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
>  92a622117c71ddfc3bd2562c04ee525491bd6ffd 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/AmbariEvent.java 
> 78731854fe80c3d6087a73ebf00b0ffe9e287354 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ClusterConfigChangedEvent.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentInstalledEvent.java
>  2ce01236b3bb1ab4934fe456e5669fa474987426 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentRecoveryChangeEvent.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentUninstalledEvent.java
>  19712cdb18592816f82898cd7aa08a3600c7e1f4 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ConfigImpl.java 
> 2cc3d0013140c9c7d3453d7c5e05c3aa02bc5794 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentImpl.java
>  4be1c21bf41416681be836716ce38305cc3f287e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
>  3a0075e3a3c90088ca6bd2d9eca48ef50db9fdb3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  b1f94e38f97550123dd366984438b0bbf9c2826c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/configuration/RecoveryConfigHelperTest.java
>  c86b95a7cd10b0720f0a3b9676f65a8bd68dd6c9 
>   
> ambari-server/src/test/java/org/apache/ambari/server/events/listeners/upgrade/HostVersionOutOfSyncListenerTest.java
>  1bf9d8336533bc67

Re: Review Request 44916: StackAdvisor needs to set config map for ams-grafana-env to meet non-null expectations



> On March 16, 2016, 5:14 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessor.java,
> >  line 224
> > 
> >
> > How can this ever be null? An empty config type!?
> 
> Robert Levas wrote:
> This is not config properties from a service config type, but updates 
> generated by the stack advisor. Up until now, this hasn't been null, but it 
> is possible based on the data the stack advisor returns.

My point was that you're adding a ton of null checks around this area of the 
code. It seems safer to maybe have the contract return an empty map? That way 
you don't have to constantly wrap this in an if-statement?


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44916/#review123930
---


On March 16, 2016, 4:55 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44916/
> ---
> 
> (Updated March 16, 2016, 4:55 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Nate Cole, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15442
> https://issues.apache.org/jira/browse/AMBARI-15442
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The StackAdvisor needs to set a config map for the `ams-grafana-env`, even it 
> is empty, to meet non-null expectations in Kerberos and Blueprint logic. 
> 
> To to this, the following line needs to be added to the StackAdvisor code:
> 
> #stacks/HDP/2.0.6/services/stack_advisor.py:473
> ```
> putGrafanaProperty = self.putProperty(configurations, "ams-grafana-env", 
> services)
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessor.java
>  ea9fffb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  be6edc9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  f5e7578 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 8c18421 
>   
> ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessorTest.java
>  c4e694b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  f6027f3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  68d5755 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> 9070c0a 
> 
> Diff: https://reviews.apache.org/r/44916/diff/
> 
> 
> Testing
> ---
> 
> Manually tested in cluster Kerbrerized using the Ambari UI.
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Review Request 44986: AMBARI-15474: Listen for changes to auto-start configuration and send them to the agent during heartbeats

2016-03-19 Thread Nahappan Somasundaram

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44986/
---

Review request for Ambari, Jonathan Hurley, Nate Cole, Sumit Mohanty, and Sid
Wagle.

Bugs: AMBARI-15474
https://issues.apache.org/jira/browse/AMBARI-15474

Repository: ambari

Description
---

AMBARI-15474: Listen for changes to auto-start configuration and send them to
the agent during heartbeats.

In HeartbeatHandler::handleHeartBeat(), send the recovery configuration to the
agent. Instead of pulling the recovery configuration from the application
cache, implement change events and get notified whenever there are changes to
the recovery configuration.

**Changes**:
1. Instead of sending the recovery configuration to the agent during every
heartbeat (10 seconds interval), send the configuration only when changes are
detected.
2. Used AmbariEventPublisher to publish the changes. RecoveryConfigHelper is
the subscriber which listens to changes to the configuration.
3. RecoveryConfigHelper maintains a map of Cluster-->Host-->Timestamp which is
the last updated timestamp of the recovery configuration. This timestamp is
used as a token and is sent to the agent with the configuration. The agent
provides this timestamp during each heartbeat. If the timestamp from the
heartbeat is the same as the one on the server side, no configuration is sent
to the agent.
4. Whenever changes are detected, the timestamp is invalidated. During the next
heartbeat, the recovery configuration is read from the DB and sent to the agent
along with the latest timestamp.
5. Added new unit tests to test each configuration change event.

Diffs
-

ambari-agent/src/main/python/ambari_agent/Controller.py
c1c16ac97cca0d2677da2e26eee7b4949a4bf15c
ambari-agent/src/main/python/ambari_agent/Heartbeat.py
b28644c834a2387d2fb0ad17d104224e830a0245
ambari-agent/src/main/python/ambari_agent/RecoveryManager.py
ed537ca6928ce376c5f3e906f7a47c3f1919e309
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeat.java
1430aa20cf68db071be85c4ad2ebb9acdaccecc0

ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
3a80803724bcb6ca9e4ec875aa17c9ac1c66fbe8

ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatResponse.java
617b04b5b8768ba28a3c8ecb6e5e00601f153396

ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfig.java
c2c18462c586292caaf40bdb6a25a8fe9a39d76c

ambari-server/src/main/java/org/apache/ambari/server/agent/RecoveryConfigHelper.java
92a622117c71ddfc3bd2562c04ee525491bd6ffd
ambari-server/src/main/java/org/apache/ambari/server/events/AmbariEvent.java
78731854fe80c3d6087a73ebf00b0ffe9e287354

ambari-server/src/main/java/org/apache/ambari/server/events/ClusterConfigChangedEvent.java
PRE-CREATION

ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentInstalledEvent.java
2ce01236b3bb1ab4934fe456e5669fa474987426

ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentRecoveryChangeEvent.java
PRE-CREATION

ambari-server/src/main/java/org/apache/ambari/server/events/ServiceComponentUninstalledEvent.java
19712cdb18592816f82898cd7aa08a3600c7e1f4
ambari-server/src/main/java/org/apache/ambari/server/state/ConfigImpl.java
2cc3d0013140c9c7d3453d7c5e05c3aa02bc5794

ambari-server/src/main/java/org/apache/ambari/server/state/ServiceComponentImpl.java
4be1c21bf41416681be836716ce38305cc3f287e

ambari-server/src/main/java/org/apache/ambari/server/state/svccomphost/ServiceComponentHostImpl.java
3a0075e3a3c90088ca6bd2d9eca48ef50db9fdb3

ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
b1f94e38f97550123dd366984438b0bbf9c2826c

ambari-server/src/test/java/org/apache/ambari/server/configuration/RecoveryConfigHelperTest.java
c86b95a7cd10b0720f0a3b9676f65a8bd68dd6c9

ambari-server/src/test/java/org/apache/ambari/server/events/listeners/upgrade/HostVersionOutOfSyncListenerTest.java
1bf9d8336533bc677384314f998f10aa2f10504e

Diff: https://reviews.apache.org/r/44986/diff/

Testing
---

**1. mvn clean install **

[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ... SUCCESS [5.864s]
[INFO] Apache Ambari Project POM . SUCCESS [0.037s]
[INFO] Ambari Web SUCCESS [23.683s]
[INFO] Ambari Views .. SUCCESS [1.069s]
[INFO] Ambari Admin View . SUCCESS [5.949s]
[INFO] ambari-metrics SUCCESS [0.534s]
[INFO] Ambari Metrics Common . SUCCESS [0.431s]
[INFO] Ambari Metrics Hadoop Sink SUCCE

Re: Review Request 44916: StackAdvisor needs to set config map for ams-grafana-env to meet non-null expectations


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44916/#review124010
---


Ship it!




My only concern is for safety and clarify of code, does it makes sense to 
change the contract of the config to always return a non-null (if empty) map?

- Jonathan Hurley


On March 16, 2016, 4:55 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44916/
> ---
> 
> (Updated March 16, 2016, 4:55 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Nate Cole, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15442
> https://issues.apache.org/jira/browse/AMBARI-15442
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The StackAdvisor needs to set a config map for the `ams-grafana-env`, even it 
> is empty, to meet non-null expectations in Kerberos and Blueprint logic. 
> 
> To to this, the following line needs to be added to the StackAdvisor code:
> 
> #stacks/HDP/2.0.6/services/stack_advisor.py:473
> ```
> putGrafanaProperty = self.putProperty(configurations, "ams-grafana-env", 
> services)
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessor.java
>  ea9fffb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  be6edc9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  f5e7578 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 8c18421 
>   
> ambari-server/src/test/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorBlueprintProcessorTest.java
>  c4e694b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  f6027f3 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
>  68d5755 
>   ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 
> 9070c0a 
> 
> Diff: https://reviews.apache.org/r/44916/diff/
> 
> 
> Testing
> ---
> 
> Manually tested in cluster Kerbrerized using the Ambari UI.
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 44916: StackAdvisor needs to set config map for ams-grafana-env to meet non-null expectations


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44916/
---

(Updated March 17, 2016, 1:48 p.m.)


Review request for Ambari, Dmytro Sen, Jonathan Hurley, Nate Cole, and Sid 
Wagle.


Bugs: AMBARI-15442
https://issues.apache.org/jira/browse/AMBARI-15442


Repository: ambari


Description
---

The StackAdvisor needs to set a config map for the `ams-grafana-env`, even it 
is empty, to meet non-null expectations in Kerberos and Blueprint logic. 

To to this, the following line needs to be added to the StackAdvisor code:

#stacks/HDP/2.0.6/services/stack_advisor.py:473
```
putGrafanaProperty = self.putProperty(configurations, "ams-grafana-env", 
services)
```


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/recommendations/RecommendationResponse.java
 4a66677 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
54d6d6e 
  ambari-server/src/test/python/stacks/2.2/common/test_stack_advisor.py 59eff4d 

Diff: https://reviews.apache.org/r/44916/diff/


Testing
---

Manually tested in cluster Kerbrerized using the Ambari UI.


Thanks,

Robert Levas

Re: Review Request 44915: https://issues.apache.org/jira/browse/HAWQ-497

2016-03-19 Thread Goutam Tadi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44915/
---

(Updated March 17, 2016, 2:28 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, jun aoki, 
and Lav Jain.


Bugs: AMBARI-15435
https://issues.apache.org/jira/browse/AMBARI-15435


Repository: ambari


Description
---

Update configs for setting virtual segments in HAWQ HashDistributed Tables and 
random distributed tables


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
 d187017 
  ambari-server/src/main/resources/common-services/HAWQ/2.0.0/themes/theme.json 
aa35311 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
8e1aff2 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 7a2a37d 

Diff: https://reviews.apache.org/r/44915/diff/


Testing
---

Yes. tested with test case


Thanks,

Goutam Tadi

Re: Review Request 44915: https://issues.apache.org/jira/browse/HAWQ-497

2016-03-19 Thread Goutam Tadi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44915/
---

(Updated March 17, 2016, 12:47 a.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, jun aoki, 
and Lav Jain.


Changes
---

tests updated


Bugs: AMBARI-15435
https://issues.apache.org/jira/browse/AMBARI-15435


Repository: ambari


Description
---

Update configs for setting virtual segments in HAWQ HashDistributed Tables and 
random distributed tables


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
 d187017 
  ambari-server/src/main/resources/common-services/HAWQ/2.0.0/themes/theme.json 
aa35311 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
8e1aff2 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 1af32d1 

Diff: https://reviews.apache.org/r/44915/diff/


Testing (updated)
---

Yes. tested with test case


Thanks,

Goutam Tadi

Re: Review Request 45041: Strange paths in ambari-server setup



> On March 18, 2016, 5:44 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/python/ambari_server/serverConfiguration.py, line 230
> > 
> >
> > This is doing rstrip instead, if the original string contains leading 
> > "/" this needs to be,
> > root = os.environ["ROOT"]
> > root = ("/" + root.lstrip("/")) if root.startswith("/") else root

Why was this dropped?


- Alejandro


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45041/#review124230
---


On March 18, 2016, 5:39 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45041/
> ---
> 
> (Updated March 18, 2016, 5:39 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Dmytro 
> Sen.
> 
> 
> Bugs: AMBARI-15483
> https://issues.apache.org/jira/browse/AMBARI-15483
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ambari-server setup command shows strange paths with double slashes in front:
> Successfully installed JDK to //usr/jdk64/
> Downloading JCE Policy archive from 
> http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip to 
> //var/lib/ambari-server/resources/jce_policy-8.zip
> Successfully downloaded JCE Policy archive to 
> //var/lib/ambari-server/resources/jce_policy-8.zip
> Installing JCE policy...
> Completing setup...
> Configuring database...
> Enter advanced database configuration y/n ?
> Configuring database...
> Default properties detected. Using built-in database.
> WARNING: Command chown None //etc/ambari-server/conf/password.dat returned 
> exit code //etc/ambari-server/conf/password.dat with message: chown: invalid 
> user: `None'
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/python/ambari_server/serverConfiguration.py 29187bd 
> 
> Diff: https://reviews.apache.org/r/45041/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Review Request 44953: Download client configurations does not work


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44953/
---

Review request for Ambari and Vitalyi Brodetskyi.


Bugs: AMBARI-15460
https://issues.apache.org/jira/browse/AMBARI-15460


Repository: ambari


Description
---

**STR**  
1) Try to download HDFS client configurations

**Result**


2016-03-16 11:53:39,160 - Using hadoop conf dir: 
/usr/hdp/current/hadoop-client/conf
Traceback (most recent call last):
  File 
"/var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py",
 line 120, in 
HdfsClient().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 219, in execute
method(env)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 605, in generate_configs
import params
  File 
"/var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py",
 line 25, in 
from params_linux import *
  File 
"/var/lib/ambari-server/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py",
 line 52, in 
agent_stack_retry_on_unavailability = 
cbool(config["hostLevelParams"]["agent_stack_retry_on_unavailability"])
  File "/usr/lib/python2.6/site-packages/ambari_commons/str_utils.py", line 
48, in cbool
return bool(obj)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
 line 81, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found 
in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 
'agent_stack_retry_on_unavailability' was not found in configurations 
dictionary!


hostLevelParams/agent_stack_retry_on_unavailability is not set.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
 648e573 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java
 3e4978a 

Diff: https://reviews.apache.org/r/44953/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 44916: StackAdvisor needs to set config map for ams-grafana-env to meet non-null expectations

2016-03-19 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44916/#review123895
---



Shouldn't we also have a non-null check in the backend code?

- Sid Wagle


On March 16, 2016, 4:59 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44916/
> ---
> 
> (Updated March 16, 2016, 4:59 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen, Jonathan Hurley, Nate Cole, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-15442
> https://issues.apache.org/jira/browse/AMBARI-15442
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The StackAdvisor needs to set a config map for the `ams-grafana-env`, even it 
> is empty, to meet non-null expectations in Kerberos and Blueprint logic. 
> 
> To to this, the following line needs to be added to the StackAdvisor code:
> 
> #stacks/HDP/2.0.6/services/stack_advisor.py:473
> ```
> putGrafanaProperty = self.putProperty(configurations, "ams-grafana-env", 
> services)
> ```
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 8c18421 
> 
> Diff: https://reviews.apache.org/r/44916/diff/
> 
> 
> Testing
> ---
> 
> Manually tested in cluster Kerbrerized using the Ambari UI.
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Review Request 44969: After upgrading from Ambari 2.1.2 to 2.2.1, HDP from 2.3.0 to 2.3.4 , user is getting 'Consistency Check Failed' error while trying to save configuration in Ranger, Hive and HBas

2016-03-19 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44969/
---

Review request for Ambari, Jaimin Jetly and Sebastian Toader.


Bugs: AMBARI-15469
https://issues.apache.org/jira/browse/AMBARI-15469


Repository: ambari


Description
---

PROBLEM:After upgrading from Ambari 2.1.2 to 2.2.1, HDP from 2.3.0 to 2.3.4 , 
user is getting 'Consistency Check Failed' error while trying to save 
configuration in Ranger, Hive and HBase

Please find the attached screenshot, and ambari-server log for more information.

{code}
Error occured in stack advisor.
Error details: list.remove(x): x not in list
14 Mar 2016 14:31:26,241  INFO [qtp-ambari-client-23360] StackAdvisorRunner:71 
- advisor script stderr: Traceback (most recent call last):
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158, in 

main(sys.argv)
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 109, in 
main
result = stackAdvisor.recommendConfigurations(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", 
line 569, in recommendConfigurations
calculation(configurations, clusterSummary, services, hosts)
File 
"/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.3/services/stack_advisor.py",
 line 239, in recommendHBASEConfigurations
super(HDP23StackAdvisor, self).recommendHBASEConfigurations(configurations, 
clusterData, services, hosts)
File 
"/var/lib/ambari-server/resources/scripts/./../stacks/HDP/2.2/services/stack_advisor.py",
 line 665, in recommendHBASEConfigurations
hbaseCoProcessorConfigs['hbase.coprocessor.master.classes'].remove('org.apache.hadoop.hbase.security.access.AccessController')
ValueError: list.remove(x): x not in list
{code}

Looks like property 'hbase.coprocessor.master.classes' is empty. The workaround 
is by putting value org.apache.hadoop.hbase.security.access.AccessController 
there.


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.2/services/stack_advisor.py 
5e23b8a 

Diff: https://reviews.apache.org/r/44969/diff/


Testing
---

mvn clean test


Thanks,

Dmitro Lisnichenko

Re: Review Request 44923: Alerts: Allow alerts to emit a structured response

2016-03-19 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44923/
---

(Updated Березень 17, 2016, 9:28 після полудня)


Review request for Ambari, Dmytro Sen and Jonathan Hurley.


Bugs: AMBARI-15445
https://issues.apache.org/jira/browse/AMBARI-15445


Repository: ambari


Description
---

For some alerts, having the ability to include a link in the response would be 
incredibly beneficial.  Having a structured response format would allow alert 
creators to add links that operators could click on to help give more context 
around an alert, or point to a remediation step.  Something like:

{code}
{
  "response": "There were more than 5 soft limits reached in the past 10 
minutes.  Please click the link to see the log file.",
  "href": "http://some.server/other/stuff";
}
{code}


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertDefinitionResourceProvider.java
 0f73ec6 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertDefinitionEntity.java
 9e21bec 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertDefinitionEntity_.java
 d2d4cf3 
  
ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinition.java
 9fff5f2 
  
ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinitionFactory.java
 3b4f5fc 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 a803f73 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql a85202d 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 9b4810c 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql cc3d197 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 07c786d 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
ab6dc93 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 8e91fde 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 440ca44 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/AlertDefinitionResourceProviderTest.java
 d44b7c8 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog240Test.java
 7f3b80c 

Diff: https://reviews.apache.org/r/44923/diff/


Testing
---

Tests will be added after patch draft review


Thanks,

Vitalyi Brodetskyi

Re: Review Request 44987: Editing tmp dir from single to multiple dir doesn't trigger the backend change


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44987/#review124115
---




ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/common.py
 (line 60)


maybe pass the mode into create_dir_as_hawq_user (default None) so it can 
be used with Directory(mode=xyz) resource and not via chmod ? Then there will 
be no need for this function at all.



ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/common.py
 (line 66)


I wonder if Ambari already supports creating a list of directories from a 
string with comma-separated values via Directory resource ?



ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/hawqsegment.py
 (line 45)


plural, so perhaps "..they should be.." instead of "..it should be.."


- Alexander Denissov


On March 17, 2016, 10:37 p.m., jun aoki wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44987/
> ---
> 
> (Updated March 17, 2016, 10:37 p.m.)
> 
> 
> Review request for Ambari.
> 
> 
> Bugs: AMBARI-15475
> https://issues.apache.org/jira/browse/AMBARI-15475
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Modified "HAWQ Segment Temp Directory” from one [/d1/tmp/hawqsegment] to many 
> [/d1/tmp/hawqsegment,/d2/tmp/hawqsegment,/d3/tmp/hawqsegment,/d4/tmp/hawqsegment,/d5/tmp/hawqsegment]
> and the change is not propagated to segments and the folders are not created.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/common.py
>  d62332d 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/hawqsegment.py
>  0a597b6 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/master_helper.py
>  9482b35 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/package/scripts/params.py
>  1bdc5aa 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqmaster.py 14b5a5f 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqsegment.py 6ca83b7 
>   ambari-server/src/test/python/stacks/2.3/HAWQ/test_hawqstandby.py eaba496 
> 
> Diff: https://reviews.apache.org/r/44987/diff/
> 
> 
> Testing
> ---
> 
> manually tested.
> 
> test_configure_default (test_hawqsegment.TestHawqSegment) ... ok
> test_install_default (test_hawqsegment.TestHawqSegment) ... ok
> test_start_default (test_hawqsegment.TestHawqSegment) ... ok
> test_stop_default (test_hawqsegment.TestHawqSegment) ... ok
> test_hawq_master_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_master_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_segment_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_segment_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_hawq_standby_critical 
> (test_alert_component_status.TestAlertComponentStatus) ... ok
> test_hawq_standby_ok (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_missing_configs (test_alert_component_status.TestAlertComponentStatus) 
> ... ok
> test_configure_default (test_hawqstandby.TestHawqStandby) ... ok
> test_install_default (test_hawqstandby.TestHawqStandby) ... ok
> test_start_default (test_hawqstandby.TestHawqStandby) ... 2016-03-17 
> 15:33:42,969 - Standby host c6401.ambari.apache.org initialized
> ok
> test_stop_default (test_hawqstandby.TestHawqStandby) ... ok
> test_configure_default (test_hawqmaster.TestHawqMaster) ... ok
> test_install_default (test_hawqmaster.TestHawqMaster) ... ok
> test_start_default (test_hawqmaster.TestHawqMaster) ... 2016-03-17 
> 15:33:43,025 - Active master c6401.ambari.apache.org initialized
> ok
> test_start_localmaster (test_hawqmaster.TestHawqMaster) ... 2016-03-17 
> 15:33:43,049 - Master c6401.ambari.apache.org started
> ok
> test_stop_default (test_hawqmaster.TestHawqMaster) ... ok
> 
> 
> Thanks,
> 
> jun aoki
> 
>

Re: Review Request 44880: AMBARI-15436 Supportability: Create 'Reset UI' button in Ambari's experimental page

2016-03-19 Thread Srimanth Gunturi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44880/#review124069
---




ambari-web/app/messages.js (line 3019)


Small grammar correction - 

Instead of:
"UI is stuck and wont loaded, etc)." 
It should be
"UI is stuck and wont load, etc.)."



ambari-web/app/templates/experimental.hbs (line 52)


The UI shows a blue button under the table which gives the impression the 
button acts on the table. My suggestion would be to have a divider () 
seperating table and button. The button should be under the divider and also in 
red color as its a dangerous action to perform.


- Srimanth Gunturi


On March 17, 2016, 1:13 a.m., Zhe (Joe) Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44880/
> ---
> 
> (Updated March 17, 2016, 1:13 a.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly, Richard Zang, Srimanth Gunturi, Xi 
> Wang, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15436
> https://issues.apache.org/jira/browse/AMBARI-15436
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently Ambari UI persists it's current state on the server. This helps 
> when a different user logs in, they end up in the same UI. Sometimes however, 
> UI's state persisted on server does not sync with cluster state, thereby 
> putting UI in a broken state with no recourse for the stuck user.
> 
> Ambari UI should provide a _Reset UI_ button in the experimental page which 
> resets the UI state on server.
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/experimental.js 9fa6da5 
>   ambari-web/app/mappers/server_data_mapper.js 1e08161 
>   ambari-web/app/messages.js 9289fc2 
>   ambari-web/app/templates/experimental.hbs 2873679 
> 
> Diff: https://reviews.apache.org/r/44880/diff/
> 
> 
> Testing
> ---
> 
> Local ambari-web test passed.
> 24578 tests complete (23 seconds)
> 145 tests pending
> Manual testing done.
> 
> 
> Thanks,
> 
> Zhe (Joe) Wang
> 
>

Re: Review Request 44923: Alerts: Allow alerts to emit a structured response


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44923/#review123914
---


Fix it, then Ship it!





ambari-server/src/main/java/org/apache/ambari/server/alerts/AgentHeartbeatAlertRunnable.java
 (lines 20 - 21)


No changes in this file; just import moving, so does it need to be in the 
commit?

Also, defaults should be:
```
java
org
com
```



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertDefinitionResourceProvider.java
 (line 560)


StringUtils.equals() is easier to read (and safer)



ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinition.java
 (line 50)


Our serialized JSON uses _ instead of camelCase (see ignoreHost).

We should add @SerializedName("help_url") here.



ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinition.java
 (lines 305 - 311)


StringUtils.equals() cuts the lines in half.



ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinitionFactory.java
 (lines 20 - 33)


Can you adjust your imports so they are:
```
java
org
com
```



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 (line 447)


You have the size defined as 512 in the entity, but 255 here.

I'm fine with either size, but we should be consistent.


- Jonathan Hurley


On March 16, 2016, 3:43 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44923/
> ---
> 
> (Updated March 16, 2016, 3:43 p.m.)
> 
> 
> Review request for Ambari, Dmytro Sen and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-15445
> https://issues.apache.org/jira/browse/AMBARI-15445
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> For some alerts, having the ability to include a link in the response would 
> be incredibly beneficial.  Having a structured response format would allow 
> alert creators to add links that operators could click on to help give more 
> context around an alert, or point to a remediation step.  Something like:
> 
> {code}
> {
>   "response": "There were more than 5 soft limits reached in the past 10 
> minutes.  Please click the link to see the log file.",
>   "href": "http://some.server/other/stuff";
> }
> {code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/alerts/AgentHeartbeatAlertRunnable.java
>  0611e23 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AlertDefinitionResourceProvider.java
>  0f73ec6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertDefinitionEntity.java
>  9e21bec 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/AlertDefinitionEntity_.java
>  d2d4cf3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinition.java
>  9fff5f2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/alert/AlertDefinitionFactory.java
>  3b4f5fc 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  a803f73 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql a85202d 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 9b4810c 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql cc3d197 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 07c786d 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> ab6dc93 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 8e91fde 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 440ca44 
> 
> Diff: https://reviews.apache.org/r/44923/diff/
> 
> 
> Testing
> ---
> 
> Tests will be added after patch draft review
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 44926: [DRAFT] Auto-retry on failure during RU/EU


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44926/#review124202
---




ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java
 (lines 334 - 340)


Documentation.



ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 (line 130)


Get rid of import.



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 51)


This class is really scoped for stack upgrades. Yet it's name doesn't 
indicate that nor is it in an upgrade package.

Thoughts on changing the name or moving its package?



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 72)


Comment on why these are skipped (for those of us who don't have upgrade 
blood in our veins)



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 84)


Consistency in name with `m_`



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 89)


Consistency in name with `m_`



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 105)


Expose via Configuration & ambari.properties



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 119)


Log.info that it's not going to run, and why.



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 133)


Use {} for log statements, especially debug ones.



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 178)


Can we fail fast for things like this?

```
if( null == requestId )
  return
```

Makes multiple if/for/if indents easier to read.



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 180)


Doesn't the contract ensure a non-null collection? You can eliminate this 
if-statement in that case.



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 207)


Doc.



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 225)


Doc



ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
 (lines 317 - 320)


Can you use named parameters here and make this a NamedQuery?


- Jonathan Hurley


On March 17, 2016, 7:07 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44926/
> ---
> 
> (Updated March 17, 2016, 7:07 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-15446
> https://issues.apache.org/jira/browse/AMBARI-15446
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a failure occurs during RU/EU and the task transitions to HOLDING_FAILED 
> or HOLDING_TIMEDOUT, want Ambari to automatically retry up to up to x mins. 
> This is useful when a host goes down as Ambari is running a task on it.
> ambari.properties will have 1 new parameter. E.g,. 
> stack-upgrade.max_retry_timeout_mins=15 (by default, will not be present)
> If Ambari Server is restarted, it should be able to recover.
> Today, Action Scheduler increases the attempt_count whenever a task is 
> retried, but it requires resetting the start_time to -1. Because of this, we 
> cannot rely on the start_time property to know when to timeout after several 
> retries.
> 
> For the implementation, will add another thread to Ambari that will monitor 
> failed tasks only during active RU/EU and change the status back to PENDING 
> so that Action Scheduler can reschedule it.
> Luckily, any tasks in HOLDING_TIMEDOUT and HOLDING_FAILED states are 
> blocking, so no other stages are allowed to proceed.
> In order to know when a task was first started, will add a new property to 
> host_role_command table called original_start_time.
> 
> For the agents, we need to ensure that the

Re: Review Request 44565: Hive view: Visual Explain does not work when there are multiple statements in the query editor

2016-03-19 Thread DIPAYAN BHOWMICK


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44565/#review124065
---


Ship it!




Ship It!

- DIPAYAN BHOWMICK


On March 17, 2016, 4:07 p.m., Pallav Kulshreshtha wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44565/
> ---
> 
> (Updated March 17, 2016, 4:07 p.m.)
> 
> 
> Review request for Ambari, DIPAYAN BHOWMICK, Srimanth Gunturi, and Yusaku 
> Sako.
> 
> 
> Bugs: AMBARI-15346
> https://issues.apache.org/jira/browse/AMBARI-15346
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Adding the "Explain | Explain Formatted" only for the last query. Make some 
> modification in "updateValue" function of query editor component to handle 
> this change.
> 
> 
> Diffs
> -
> 
>   
> contrib/views/hive/src/main/resources/ui/hive-web/app/components/query-editor.js
>  10c6bb3 
>   contrib/views/hive/src/main/resources/ui/hive-web/app/controllers/index.js 
> a056759 
> 
> Diff: https://reviews.apache.org/r/44565/diff/
> 
> 
> Testing
> ---
> 
> manual testing done. No extra unit test case needed.
> 
> 
> Thanks,
> 
> Pallav Kulshreshtha
> 
>

Re: Review Request 44725: After exporting blueprint from ranger enabled cluster ranger.service.https.attrib.keystore.pass is exported

2016-03-19 Thread Amruta Borkar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44725/
---

(Updated March 17, 2016, 6:52 p.m.)


Review request for Ambari, Di Li and Robert Nettleton.


Changes
---

Currently Secret References are getting filtered based on Property Name, but as 
properties are ending with difference strings (password, secret, pass), the 
filtering expression needs to be modified every time. 

As these password properties have values that follow the same format 
"SECRET:property_name:version:property_name" we can filter Secret References 
based on property value, rather than property name, as this would make it 
generic.


Bugs: AMBARI-15338
https://issues.apache.org/jira/browse/AMBARI-15338


Repository: ambari


Description
---

After exporting blueprint from ranger enabled cluster 
ranger.service.https.attrib.keystore.pass is exported.
Which needs to be removed before using the same blueprint to create another 
cluster
Error Show when used same blueprint:
{ "status" : 400, "message" : "Blueprint configuration validation failed: 
Secret references are not allowed in blueprints, replace following properties 
with real passwords:\n Config:ranger-admin-site 
Property:ranger.service.https.attrib.keystore.pass\n" }


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
 f5e7578 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java
 68d5755 

Diff: https://reviews.apache.org/r/44725/diff/


Testing
---

Modified test cases to test for if the properties that end with "pass" are 
getting filtered. Other properties which have 'pass' else where in the name 
will not get filtered.


Thanks,

Amruta Borkar

Re: Review Request 44878: Atlas Integration : Rename Atlas Configurations


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44878/#review123865
---


Ship it!




Ship It!

- Nate Cole


On March 15, 2016, 8:57 p.m., Tom Beerbower wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44878/
> ---
> 
> (Updated March 15, 2016, 8:57 p.m.)
> 
> 
> Review request for Ambari, John Speidel, Nate Cole, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-15431
> https://issues.apache.org/jira/browse/AMBARI-15431
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Atlas configuration name application.properties has been changed to 
> atlas-application.properties to avoid name conflicts with other services. 
> See https://issues.apache.org/jira/browse/ATLAS-392.
> 
> Ambari scripts for Atlas currently use the configuration name 
> application.properties for all stack levels. Stacks which include Atlas > 0.5 
> should use the configuration name atlas-application.properties.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/configuration/atlas-env.xml
>  8500488 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
>  6df47b0 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py
>  5a39278 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
>  38c2c9b 
>   
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/status_params.py
>  c402721 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon.py
>  b131574 
>   
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py
>  115ed4f 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py
>  ec37243 
>   
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop.py
>  da67775 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-env.xml
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/metainfo.xml 
> 9514576 
>   ambari-server/src/test/python/stacks/2.3/configs/default.json d5102dc 
>   ambari-server/src/test/python/stacks/2.3/configs/secure.json 2b60be6 
>   ambari-server/src/test/python/stacks/2.6/ATLAS/test_atlas_server.py 
> PRE-CREATION 
>   ambari-server/src/test/python/stacks/2.6/configs/default.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/44878/diff/
> 
> 
> Testing
> ---
> 
> Manual test install Atlas (HDP 2.6 stack).  Verify configuration.
> 
> mvn clean test
> 
> all pass
> 
> 
> Thanks,
> 
> Tom Beerbower
> 
>

Re: Review Request 44941: Report AMS / Grafana Per Disk Metrics and other changes

2016-03-19 Thread Sumit Mohanty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44941/#review123978
---




ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/host_info.py 
(line 204)


nice :-)



ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/host_info.py 
(line 222)


delete these - any reason to comment them


- Sumit Mohanty


On March 17, 2016, 1:09 a.m., Sid Wagle wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44941/
> ---
> 
> (Updated March 17, 2016, 1:09 a.m.)
> 
> 
> Review request for Ambari, Aravindan Vijayan, Dmytro Sen, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-15448
> https://issues.apache.org/jira/browse/AMBARI-15448
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> We need to collect and expose per-disk metrics from AMS for iostats counters.
> We can sort the disk names alphabetically, and name the metrics like _1, _2, 
> etc.
> 
> Plus from psutils:
> busy_time: (Linux, FreeBSD) time spent doing actual I/Os (in milliseconds)
> read_merged_count (Linux): number of merged reads (see iostat doc)
> write_merged_count (Linux): number of merged writes (see iostats doc)
> 
> _Additional fixes_:
> - Combined_disk_total metric seems to be wrong : it is a sum() of all disk 
> percentages from different disks
> - Missing ATS and JHS metrics due to wrong config item in 
> hadoop-metrics2.properties file
> - Report number of disks
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/config_reader.py
>  a053955 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/controller.py
>  c04a61b 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/host_info.py
>  2e8c442 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/main/python/core/metric_collector.py
>  c28fd03 
>   
> ambari-metrics/ambari-metrics-host-monitoring/src/test/python/core/TestHostInfo.py
>  f07b573 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_grafana_util.py
>  e5020a5 
>   
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/templates/metric_monitor.ini.j2
>  0b0932a 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/templates/hadoop-metrics2.properties.j2
>  47b504f 
> 
> Diff: https://reviews.apache.org/r/44941/diff/
> 
> 
> Testing
> ---
> 
> Manually verified.
> 
> --
> Ran 12 tests in 0.020s
> 
> OK
> --
> Total run:12
> Total errors:0
> Total failures:0
> OK
> 
> 
> Thanks,
> 
> Sid Wagle
> 
>

Re: Review Request 44835: AMBARI-15420: Refactor resource_management library


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/
---

(Updated March 18, 2016, 8:57 p.m.)


Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo  
Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-15420
https://issues.apache.org/jira/browse/AMBARI-15420


Repository: ambari


Description
---

Refactor resource_management library, to generalize the library and remove 
HDP-specific hardcodings

Specifically,
1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding 
hdp-select and conf-select
2. Make stack-root stack-driven instead of hardcoding /usr/hdp
3. Make copy_tarball mappings stack-driven 
4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of 
hardcoding the "/usr/hdp"

In addition, also added a feature in the stack processing engine to support 
properties values to be defined a external property file (See tarball_map.json, 
stack_tools.json in patch)

Three config properties are added
1. cluster-env/stack_root
2. cluster-env/stack_tools
3. cluster-env/tarball_map
Corresponding helper functions get_stack_root(), get_stack_tool(), 
get_tarball_map() are added in script.py, which will set the defaults if these 
config properties are not defined (ambari-server upgrade scenario needs to be 
addressed to add these config properties on upgrade). These helper functions 
are used to remove hardcodings in resource_management library.


Remaining HDP-specific logic in resource_management library

1. conf_select::_valid()
2. conf_select::get_hadoop_conf_dir()
3. list_ambari_managed_repos::repository_names
4. version_select_util::get_component_version()
5. script::get_stack_version()
6. script::should_expose_component_version()
7. get_lzo_packages::get_lzo_packages()

Refactoring this HDP-specific logic would require "Stack Featurization" 
(AMBARI-13364) to be in place.


Diffs
-

  ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py 
ee7db0a 
  ambari-common/src/main/python/resource_management/core/utils.py 71d0008 
  
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
 770595f 
  
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
 647b8b6 
  
ambari-common/src/main/python/resource_management/libraries/functions/default.py
 23383dc 
  
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
 a20b03c 
  
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
 f2e6567 
  
ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py
 f1cd9cb 
  
ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py
 d585dea 
  
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py
 4d9d8a4 
  
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
 2ccc0c6 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
 4a8eeb9 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py
 95c5cba 
  ambari-common/src/main/python/resource_management/libraries/script/script.py 
1b0116e 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 4be4049 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java
 7f21aaa 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
0c7faea 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java
 c739211 
  ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
7d934bb 
  ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java 
e7c9c27 
  
ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java
 3f7f756 
  ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java 
PRE-CREATION 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py
 12ca388 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py
 45dbb24 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
 d000846 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py
 2c06c40 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py
 535c821 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py
 85e7012 
  
ambari-server

Review Request 44957: [Ambari tarballs] ambari-server setup fails


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44957/
---

Review request for Ambari and Vitalyi Brodetskyi.


Bugs: AMBARI-15464
https://issues.apache.org/jira/browse/AMBARI-15464


Repository: ambari


Description
---

[ambari@c6401 ~]$ /opt/usr/sbin/ambari-server status
Using python  /usr/bin/python
Ambari-server status
Traceback (most recent call last):
  File "/opt/usr/sbin/ambari-server.py", line 33, in 
from ambari_server.dbConfiguration import DATABASE_NAMES, 
LINUX_DBMS_KEYS_LIST
  File 
"/opt/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration.py", line 
28, in 
from ambari_server.serverConfiguration import 
decrypt_password_for_alias, get_ambari_properties, get_is_secure, \
  File 
"/opt/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", 
line 517, in 
configDefaults = ServerConfigDefaults()
  File 
"/opt/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", 
line 438, in __init__
super(ServerConfigDefaultsLinux, self).__init__()
  File 
"/opt/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", 
line 328, in __init__
self.OUT_DIR = 
parse_log4j_file("/etc/ambari-server/conf/log4j.properties")['ambari.log.dir']
  File "/opt/usr/lib/python2.6/site-packages/ambari_commons/os_utils.py", 
line 188, in parse_log4j_file
with open(filename, "rb") as fp:
IOError: [Errno 2] No such file or directory: 
'/etc/ambari-server/conf/log4j.properties'


Diffs
-

  ambari-server/src/main/python/ambari_server/serverConfiguration.py dc1fd8a 

Diff: https://reviews.apache.org/r/44957/diff/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 44915: https://issues.apache.org/jira/browse/HAWQ-497


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44915/#review123956
---




ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py (line 
691)


did you run the tests ? I believe test_stack_advisor.py will need to get 
updated.


- Alexander Denissov


On March 16, 2016, 9:46 p.m., Goutam Tadi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44915/
> ---
> 
> (Updated March 16, 2016, 9:46 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, jun aoki, 
> and Lav Jain.
> 
> 
> Bugs: AMBARI-15435
> https://issues.apache.org/jira/browse/AMBARI-15435
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Update configs for setting virtual segments in HAWQ HashDistributed Tables 
> and random distributed tables
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/configuration/hawq-site.xml
>  d3aa0ed 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/themes/theme.json 
> aa35311 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 8e1aff2 
> 
> Diff: https://reviews.apache.org/r/44915/diff/
> 
> 
> Testing
> ---
> 
> Yes. Manually.
> 
> 
> Thanks,
> 
> Goutam Tadi
> 
>

Re: Review Request 45041: Strange paths in ambari-server setup


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45041/#review124233
---


Ship it!




Ship It!

- Andrew Onischuk


On March 18, 2016, 5:39 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45041/
> ---
> 
> (Updated March 18, 2016, 5:39 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Dmitro Lisnichenko, and Dmytro 
> Sen.
> 
> 
> Bugs: AMBARI-15483
> https://issues.apache.org/jira/browse/AMBARI-15483
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> ambari-server setup command shows strange paths with double slashes in front:
> Successfully installed JDK to //usr/jdk64/
> Downloading JCE Policy archive from 
> http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip to 
> //var/lib/ambari-server/resources/jce_policy-8.zip
> Successfully downloaded JCE Policy archive to 
> //var/lib/ambari-server/resources/jce_policy-8.zip
> Installing JCE policy...
> Completing setup...
> Configuring database...
> Enter advanced database configuration y/n ?
> Configuring database...
> Default properties detected. Using built-in database.
> WARNING: Command chown None //etc/ambari-server/conf/password.dat returned 
> exit code //etc/ambari-server/conf/password.dat with message: chown: invalid 
> user: `None'
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/python/ambari_server/serverConfiguration.py 29187bd 
> 
> Diff: https://reviews.apache.org/r/45041/diff/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 44835: AMBARI-15420: Refactor resource_management library


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44835/
---

(Updated March 18, 2016, 7:26 p.m.)


Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Juanjo  
Marron, Nate Cole, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-15420
https://issues.apache.org/jira/browse/AMBARI-15420


Repository: ambari


Description
---

Refactor resource_management library, to generalize the library and remove 
HDP-specific hardcodings

Specifically,
1. Make stack-selector, conf-selector tools stack-driven instead of hardcoding 
hdp-select and conf-select
2. Make stack-root stack-driven instead of hardcoding /usr/hdp
3. Make copy_tarball mappings stack-driven 
4. Make PACKAGE_DIRS mapping in conf_select use stack-root instead of 
hardcoding the "/usr/hdp"

In addition, also added a feature in the stack processing engine to support 
properties values to be defined a external property file (See tarball_map.json, 
stack_tools.json in patch)

Three config properties are added
1. cluster-env/stack_root
2. cluster-env/stack_tools
3. cluster-env/tarball_map
Corresponding helper functions get_stack_root(), get_stack_tool(), 
get_tarball_map() are added in script.py, which will set the defaults if these 
config properties are not defined (ambari-server upgrade scenario needs to be 
addressed to add these config properties on upgrade). These helper functions 
are used to remove hardcodings in resource_management library.


Remaining HDP-specific logic in resource_management library

1. conf_select::_valid()
2. conf_select::get_hadoop_conf_dir()
3. list_ambari_managed_repos::repository_names
4. version_select_util::get_component_version()
5. script::get_stack_version()
6. script::should_expose_component_version()
7. get_lzo_packages::get_lzo_packages()

Refactoring this HDP-specific logic would require "Stack Featurization" 
(AMBARI-13364) to be in place.


Diffs (updated)
-

  ambari-agent/src/main/python/ambari_agent/HostCheckReportFileHandler.py 
ee7db0a 
  ambari-common/src/main/python/resource_management/core/utils.py 71d0008 
  
ambari-common/src/main/python/resource_management/libraries/functions/conf_select.py
 770595f 
  
ambari-common/src/main/python/resource_management/libraries/functions/copy_tarball.py
 647b8b6 
  
ambari-common/src/main/python/resource_management/libraries/functions/default.py
 23383dc 
  
ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py
 a20b03c 
  
ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py
 f2e6567 
  
ambari-common/src/main/python/resource_management/libraries/functions/install_windows_msi.py
 f1cd9cb 
  
ambari-common/src/main/python/resource_management/libraries/functions/repo_version_history.py
 d585dea 
  
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin.py
 4d9d8a4 
  
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
 2ccc0c6 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py
 4a8eeb9 
  
ambari-common/src/main/python/resource_management/libraries/functions/stack_tools.py
 PRE-CREATION 
  
ambari-common/src/main/python/resource_management/libraries/functions/version_select_util.py
 95c5cba 
  ambari-common/src/main/python/resource_management/libraries/script/script.py 
1b0116e 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 4be4049 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ConfigurationDirectory.java
 7f21aaa 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
0c7faea 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackDefinitionDirectory.java
 c739211 
  ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
7d934bb 
  ambari-server/src/main/java/org/apache/ambari/server/state/PropertyInfo.java 
e7c9c27 
  
ambari-server/src/main/java/org/apache/ambari/server/state/ValueAttributesInfo.java
 3f7f756 
  ambari-server/src/main/java/org/apache/ambari/server/utils/JsonUtils.java 
PRE-CREATION 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py
 12ca388 
  
ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/status_params.py
 45dbb24 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py
 d000846 
  
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/status_params.py
 2c06c40 
  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/status_params.py
 535c821 
  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py
 85e7012 
  
amb

Re: Review Request 44926: [DRAFT] Auto-retry on failure during RU/EU


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44926/#review124117
---




ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java
 (line 61)


Today, startTime is allowed to be changed to -1 after it already had value. 
For this reason, I added a column called originalStartTime that can only be set 
once.



ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
 (line 51)


Switched to guava service :-)



ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 (line 1123)


Feature will be turned off by default.



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 (line 643)


Added column on ambari upgrade.


- Alejandro Fernandez


On March 17, 2016, 11:07 p.m., Alejandro Fernandez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44926/
> ---
> 
> (Updated March 17, 2016, 11:07 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-15446
> https://issues.apache.org/jira/browse/AMBARI-15446
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When a failure occurs during RU/EU and the task transitions to HOLDING_FAILED 
> or HOLDING_TIMEDOUT, want Ambari to automatically retry up to up to x mins. 
> This is useful when a host goes down as Ambari is running a task on it.
> ambari.properties will have 1 new parameter. E.g,. 
> stack-upgrade.max_retry_timeout_mins=15 (by default, will not be present)
> If Ambari Server is restarted, it should be able to recover.
> Today, Action Scheduler increases the attempt_count whenever a task is 
> retried, but it requires resetting the start_time to -1. Because of this, we 
> cannot rely on the start_time property to know when to timeout after several 
> retries.
> 
> For the implementation, will add another thread to Ambari that will monitor 
> failed tasks only during active RU/EU and change the status back to PENDING 
> so that Action Scheduler can reschedule it.
> Luckily, any tasks in HOLDING_TIMEDOUT and HOLDING_FAILED states are 
> blocking, so no other stages are allowed to proceed.
> In order to know when a task was first started, will add a new property to 
> host_role_command table called original_start_time.
> 
> For the agents, we need to ensure that they always write out a response. On 
> the first heartbeat, it should send the status of its last command so we know 
> it failed and Ambari can retry.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionDBAccessorImpl.java
>  429f573 
>   
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java
>  2764b3f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  3a80803 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a1a686a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/RetryActionMonitor.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  9404506 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostRoleCommandDAO.java
>  f5b1cb4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostRoleCommandEntity.java
>  19f0602 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java
>  9eb514a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/LogicalRequest.java
>  82edbcf 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
>  a803f73 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql 9b4810c 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql cc3d197 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 07c786d 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-EMBEDDED-CREATE.sql 
> ab6dc93 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 8e91fde 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 440ca44 
> 
> Diff: https://reviews.apache.org/r/44926/diff/
> 
> 
> Testing
> ---
> 
> Verified on a live cluster.
> 
> TODO: Still need to make more changes to the implementation, add the config, 
> switch to gauva service, add a column, and add unit tests.
> 
> 
> Thanks,
> 
> Alejandro Fernandez
> 
>

Re: Review Request 44958: Increase Ambari Server Perm gen default value