Review Request 60553: AMBARI-21378. BigInsights stack changes needed to make Ambari upgrade functional by repointing Flume and Oozie

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60553/
---

Review request for Ambari, Di Li, Jonathan Hurley, Sid Wagle, and Tim Thorpe.


Bugs: AMBARI-21378
https://issues.apache.org/jira/browse/AMBARI-21378


Repository: ambari


Description
---

Ambari Upgrade from BigInsights stack 4.2.5 to Ambari 2.5.next fails if Flume 
and Oozie are present since the version it is pointing does not exist in 
common-services.

FLUME: Re-point to 1.4.2.0 definition
OOZIE: Re-point to 4.2.0.2.5 definition


Diffs
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/FLUME/metainfo.xml
 bed2822 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/JNBG/metainfo.xml
 3520a32 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/metainfo.xml
 2e74c56 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/R4ML/metainfo.xml
 d1c708d 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SOLR/metainfo.xml
 be20810 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/SYSTEMML/metainfo.xml
 7a0e125 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/TITAN/metainfo.xml
 d00e707 


Diff: https://reviews.apache.org/r/60553/diff/1/


Testing
---

Verified on local cluster


Thanks,

Alejandro Fernandez

Re: Review Request 60480: AMBARI-21351 Add Support For MariaDB to Python Scripts

[Di Li]

> On June 28, 2017, 7:51 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml
> > Lines 118 (patched)
> > 
> >
> > This is missing 
> 
> Di Li wrote:
> right, I removed it per Jonathan's comment. Or I can add it back but set 
> to false ( not seeing the point of that comparing to not having the line at 
> all)
> 
> " This will cause upgrades to change configs and have Hive report 
> staleness and require a restart. Instead, it seems like we should default to 
> False if not found and use the upgrade packs to update this.
> 
> In other words; since Stack-A might need this when moving to Stack-B, 
> then it should provide this as a configuration setting. "
> 
> Di Li wrote:
> I think I misundertood Jonathan's comment. It should have been a Py only 
> change with default value to false and rely on config update during the 
> migration to tell Py to use mariadb instead of mysqld.
> 
> Jonathan Hurley wrote:
> Yes, my point was to track this in trunk. A cluster which has gone 
> through an upgrade with Ambari 2.5 will regress on Ambari 3.0. The problem is 
> that we should really be tracking this via a real config property. 
> Introducing the property is fine, but on-ambari-upgrade=false must be used 
> since it's not critical for Ambari's operation. This means it should have 
> sensible defaults and should be updated using the upgrade packs for stack 
> upgrade.
> 
> The whole problem centers around downtime. If a business has an outage 
> for an Ambari upgrade, they won't be able to restart Hive when it shows stale 
> configs.

How about the latest patch where the only change is in the Python script 
(default to false of the special property). It's then up to the upgrade pack or 
preupgrad manual step to set the value if cluster is a migrated one? 

This patch can then be applied to both the feature branch and trunk.


- Di


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60480/#review179157
---


On June 29, 2017, 3:47 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60480/
> ---
> 
> (Updated June 29, 2017, 3:47 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21351
> https://issues.apache.org/jira/browse/AMBARI-21351
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Some stacks, such as BigInights, support MariaDB as the database for Hive. 
> When upgrading from BigInights to another stack, such as HDP, this causes the 
> following exception when starting the Hive service:
> ```resource_management.core.exceptions.ExecutionFailed: Execution of 'service 
> mysqld stop' returned 5. Redirecting to /bin/systemctl stop  mysqld.service
> Failed to stop mysqld.service: Unit mysqld.service not loaded.```
> This looks to be caused by the hard-coded reliance of mysqld in the Python 
> scripts.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py
>  b7cb148 
> 
> 
> Diff: https://reviews.apache.org/r/60480/diff/4/
> 
> 
> Testing
> ---
> 
> Tim and I patched a cluster with basically the same fix when migrating an IOP 
> 4.2.5 cluster to HDP 2.6/Ambari 2.5.1. THis patch has one additional change 
> that's to set StackFeature boolean in case this code gets merged back to 
> trunk later on. 
> 
> If the code only stays in this branch, the StackFeature boolean change can be 
> omitted.
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

[Jonathan Hurley]

> On June 29, 2017, 1:13 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml
> > Lines 232 (patched)
> > 
> >
> > Same change is needed in trunk for Hive in 3.0

Ah - right - good catch... I keep forgetting we have essentially 2 versions of 
every service in trunk now.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/#review179277
---


On June 29, 2017, 12:28 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60543/
> ---
> 
> (Updated June 29, 2017, 12:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21364
> https://issues.apache.org/jira/browse/AMBARI-21364
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129
> # Upgrade Ambari to 2.5.2.0-74 (hash: 
> fd30644590991deb41241454d6e9091ed7a38e92)
> # Run "ambari-server start"
> 
> {code}
> root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
> restart
> Using python  /usr/bin/python
> Restarting ambari-server
> Waiting for server stop...
> Ambari Server stopped
> Ambari Server running with administrator privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Ambari database consistency check started...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Waiting for server start..
> DB configs consistency check failed. Run "ambari-server start 
> --skip-database-check" to skip. You may try --auto-fix-database flag to 
> attempt to fix issues automatically. If you use this "--skip-database-check" 
> option, do not make any changes to your cluster topology or perform a cluster 
> upgrade until you correct the database consistency issues. See 
> /var/log/ambari-server/ambari-server-check-database.log for more details on 
> the consistency issues.
> ERROR: Exiting with exit code -1.
> REASON: Ambari Server java process has stopped. Please check the logs for 
> more information.
> {code}
> 
> DB log: ambari-server-check-database.log
> {code}
> 2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
> 2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
> configs that we got from db
> 2017-06-27 13:51:38,748  INFO - Getting services from metainfo
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KNOX
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KERBEROS
> 2017-06-27 13:51:38,749  INFO - Comparing required service configs from stack 
> with mapped service configs from db
> 2017-06-27 13:51:38,751 ERROR - Required config(s): product-info is(are) not 
> available for service SMARTSENSE with service config version 2 in cluster cl1
> 2017-06-27 13:51:38,751 ERROR - Required config(s): parquet-logging is(are) 
> not available for service 

Re: Review Request 60480: AMBARI-21351 Add Support For MariaDB to Python Scripts

[Jonathan Hurley]

> On June 28, 2017, 3:51 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml
> > Lines 118 (patched)
> > 
> >
> > This is missing 
> 
> Di Li wrote:
> right, I removed it per Jonathan's comment. Or I can add it back but set 
> to false ( not seeing the point of that comparing to not having the line at 
> all)
> 
> " This will cause upgrades to change configs and have Hive report 
> staleness and require a restart. Instead, it seems like we should default to 
> False if not found and use the upgrade packs to update this.
> 
> In other words; since Stack-A might need this when moving to Stack-B, 
> then it should provide this as a configuration setting. "
> 
> Di Li wrote:
> I think I misundertood Jonathan's comment. It should have been a Py only 
> change with default value to false and rely on config update during the 
> migration to tell Py to use mariadb instead of mysqld.

Yes, my point was to track this in trunk. A cluster which has gone through an 
upgrade with Ambari 2.5 will regress on Ambari 3.0. The problem is that we 
should really be tracking this via a real config property. Introducing the 
property is fine, but on-ambari-upgrade=false must be used since it's not 
critical for Ambari's operation. This means it should have sensible defaults 
and should be updated using the upgrade packs for stack upgrade.

The whole problem centers around downtime. If a business has an outage for an 
Ambari upgrade, they won't be able to restart Hive when it shows stale configs.


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60480/#review179157
---


On June 29, 2017, 11:47 a.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60480/
> ---
> 
> (Updated June 29, 2017, 11:47 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21351
> https://issues.apache.org/jira/browse/AMBARI-21351
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Some stacks, such as BigInights, support MariaDB as the database for Hive. 
> When upgrading from BigInights to another stack, such as HDP, this causes the 
> following exception when starting the Hive service:
> ```resource_management.core.exceptions.ExecutionFailed: Execution of 'service 
> mysqld stop' returned 5. Redirecting to /bin/systemctl stop  mysqld.service
> Failed to stop mysqld.service: Unit mysqld.service not loaded.```
> This looks to be caused by the hard-coded reliance of mysqld in the Python 
> scripts.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py
>  b7cb148 
> 
> 
> Diff: https://reviews.apache.org/r/60480/diff/4/
> 
> 
> Testing
> ---
> 
> Tim and I patched a cluster with basically the same fix when migrating an IOP 
> 4.2.5 cluster to HDP 2.6/Ambari 2.5.1. THis patch has one additional change 
> that's to set StackFeature boolean in case this code gets merged back to 
> trunk later on. 
> 
> If the code only stays in this branch, the StackFeature boolean change can be 
> omitted.
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 60537: AMBARI-21373 Remove clean repo logic not suitable for Apache Ambari Python API

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60537/#review179292
---


Ship it!




Ship It!

- Jonathan Hurley


On June 29, 2017, 9:39 a.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60537/
> ---
> 
> (Updated June 29, 2017, 9:39 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21373
> https://issues.apache.org/jira/browse/AMBARI-21373
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The  clean repo logic is internal only, remove it in Apache Ambari as 
> community version does not have necessary Python API in recourse_management. 
> Also the code is not a must-have
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/hooks/before-INSTALL/scripts/repo_initialization.py
>  1b0f5c6 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-INSTALL/scripts/repo_initialization.py
>  98fa131 
> 
> 
> Diff: https://reviews.apache.org/r/60537/diff/1/
> 
> 
> Testing
> ---
> 
> Patched an IOP/HDP cluster post Ambari upgrade, before migration.
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 60540: AMBARI-21339 logviewer started along with nimbus if supervisor is not running on the same machine

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60540/#review179290
---


Ship it!




Ship It!

- Sebastian Toader


On June 29, 2017, 5:20 p.m., Laszlo Puskas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60540/
> ---
> 
> (Updated June 29, 2017, 5:20 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Sumit Mohanty, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21339
> https://issues.apache.org/jira/browse/AMBARI-21339
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Logviewer is only started along with the supervisor so inn cases when nimbus 
> runs on a node not hosting supervisor logviever doesn't run.
> 
> Solution:
> - in such cases the logviewr is started along with nimbus
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus.py
>  a6bc437 
> 
> 
> Diff: https://reviews.apache.org/r/60540/diff/1/
> 
> 
> Testing
> ---
> 
> Manually on dev env.
> 
> - provision cluster with storm, nimbus and supervisor running on the same 
> host -> logviewer started along with *supervisor*
> - provision cluster with storm, nimbus running "alone" on the node -> 
> logviewer started along with *nimbus*
> 
> 
> Thanks,
> 
> Laszlo Puskas
> 
>

Re: Review Request 60535: AMBARI-21372. Allow Dependency Checks to be Skipped During Stack Package Installation

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60535/#review179288
---




ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
Lines 413 (patched)


This name may be too generic, let's call it "ignore_package_dependencies"



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
Lines 116 (patched)


Is this param going to be passed in via the UI?

Perhaps this file should always set it to true if the current stack and the 
stack to install differ.


- Alejandro Fernandez


On June 29, 2017, 12:55 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60535/
> ---
> 
> (Updated June 29, 2017, 12:55 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Sumit 
> Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21372
> https://issues.apache.org/jira/browse/AMBARI-21372
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Introduce `ignore_dependencies` flag for `ClusterStackVersions` request.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
>  edf12514a2966d4f29877e93c702f107ed83f37a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
>  621d71e9391200be057d1efd6657dd3562d459bb 
>   ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
> 3a93389957e36686848217fd7f69b8af0101d174 
> 
> 
> Diff: https://reviews.apache.org/r/60535/diff/1/
> 
> 
> Testing
> ---
> 
> Manual test on CentOS7.
> 
> With `ignore_dependencies: true`:
> 
> ```
> $ curl -X POST -d '{ "ClusterStackVersions": { "stack": "HDP", "version": 
> "2.6", "repository_version": "2.6.3.0-11", "ignore_dependencies": "true" } }' 
> http://localhost:8080/api/v1/clusters/TEST/stack_versions
> 
> output-52.txt:
> ...
> 2017-06-29 11:57:27,098 - Will install packages for repository version 
> 2.6.3.0-11
> ...
> 2017-06-29 11:57:27,133 - Installing package hdp-select ('/usr/bin/yum -d 0 
> -e 0 -y install hdp-select')
> ...
> 2017-06-29 11:58:58,669 - Installing package zookeeper_2_6_3_0_11-server 
> ('/usr/bin/yum -d 0 -e 0 -y install zookeeper_2_6_3_0_11-server')
> 2017-06-29 11:59:30,007 - Ignoring package dependencies
> ...
> 2017-06-29 11:59:30,028 - Configuration symlinks are not needed for 2.6.3.0-11
> ```
> 
> Defaults to `false` if flag is omitted:
> 
> ```
> $ curl -X POST -d '{ "ClusterStackVersions": { "stack": "HDP", "version": 
> "2.6", "repository_version": "2.6.3.0-11" } }' 
> http://localhost:8080/api/v1/clusters/TEST/stack_versions
> 
> output-52.txt:
> ...
> 2017-06-29 12:03:02,930 - Will install packages for repository version 
> 2.6.3.0-11
> ...
> 2017-06-29 12:03:02,999 - Installing package hdp-select ('/usr/bin/yum -d 0 
> -e 0 -y install hdp-select')
> ...
> 2017-06-29 12:04:29,980 - Installing package zookeeper_2_6_3_0_11-server 
> ('/usr/bin/yum -d 0 -e 0 -y install zookeeper_2_6_3_0_11-server')
> 2017-06-29 12:05:08,189 - checked_call[['/usr/bin/yum', '-d', '0', '-e', '0', 
> 'check', 'dependencies']] {'sudo': True}
> 2017-06-29 12:05:08,953 - Could not install packages. Error: Execution of 
> '/usr/bin/yum -d 0 -e 0 check dependencies' returned 1. Error: check 
> ['dependencies']
> ...
> ```
> 
> 
> Unit tests (same as on `branch-feature-AMBARI-21348`):
> 
> ```
> Failed tests:
>   UpgradeActionTest.testFinalizeDowngrade:515 expected: but 
> was:
>   UpgradeActionTest.testFinalizeDowngradeAcrossStacks:813 expected:<4> but 
> was:<8>
>   UpgradeActionTest.testFinalizeUpgrade:606->verifyBaseRepoURL:674 
> expected:<1> but was:<0>
>   ConfigUpgradeValidityTest.testValidateConfigUpgradePacks:225 File 
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  appears to be a config upgrade pack, but does not define 
> 'upgrade-config.xsd' as its schema
> Tests in error:
>   
> UpgradeActionTest.testFinalizeDowngradeWhenDidNotFinalizePreviousUpgrade:541->makeTwoUpgradesWhereLastDidNotComplete:254
>  » Ambari
>   ServicePropertiesTest.validatePropertySchemaOfServiceXMLs:50 » Ambari File 
> /Us...
> 
> Tests run: 4993, Failures: 4, Errors: 2, Skipped: 39
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 60541: Using URI naming format in dfs.datanode.data.dir causing datanodes startup failure

[Vitalyi Brodetskyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60541/#review179279
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Червень 29, 2017, 3:18 після полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60541/
> ---
> 
> (Updated Червень 29, 2017, 3:18 після полудня)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21376
> https://issues.apache.org/jira/browse/AMBARI-21376
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Using URI naming format in dfs.datanode.data.dir causing datanodes startup
> failure.  
> I can replicate the issue locally @  master1.openstacklocal:8080/>  
> See task logs for starting DN for data1.openstacklocal / data2.openstacklocal
> /data3.openstacklocal
> 
> 
> 
> 
> 2017-06-26 15:30:52,442 - Using hadoop conf dir: 
> /usr/hdp/current/hadoop-client/conf
> 2017-06-26 15:30:52,603 - Stack Feature Version Info: stack_version=2.5, 
> version=2.5.3.0-37, current_cluster_version=2.5.3.0-37 -> 2.5.3.0-37
> 2017-06-26 15:30:52,604 - Using hadoop conf dir: 
> /usr/hdp/current/hadoop-client/conf
> User Group mapping (user_group) is missing in the hostLevelParams
> 2017-06-26 15:30:52,605 - Group['livy'] {}
> 2017-06-26 15:30:52,607 - Group['spark'] {}
> 2017-06-26 15:30:52,607 - Group['hadoop'] {}
> 2017-06-26 15:30:52,607 - Group['users'] {}
> 2017-06-26 15:30:52,607 - Group['knox'] {}
> 2017-06-26 15:30:52,608 - User['hive'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,609 - User['zookeeper'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,609 - User['ams'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,610 - User['tez'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2017-06-26 15:30:52,611 - User['livy'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,611 - User['spark'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,612 - User['ambari-qa'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2017-06-26 15:30:52,612 - User['hdfs'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,613 - User['yarn'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,614 - User['mapred'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,614 - User['hbase'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,615 - User['knox'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,616 - User['hcat'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,616 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] 
> {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2017-06-26 15:30:52,618 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh 
> ambari-qa 
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
>  {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
> 2017-06-26 15:30:52,628 - Skipping 
> Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa 
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
>  due to not_if
> 2017-06-26 15:30:52,628 - Directory['/tmp/hbase-hbase'] {'owner': 
> 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
> 2017-06-26 15:30:52,634 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] 
> {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2017-06-26 15:30:52,635 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh 
> hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] 
> {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
> 2017-06-26 15:30:52,645 - Skipping 
> Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase 
> /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due 
> to not_if
> 2017-06-26 15:30:52,646 - Group['hdfs'] {}
> 2017-06-26 15:30:52,647 - User['hdfs'] {'fetch_nonlocal_groups': True, 
> 'groups': ['hadoop', 'hdfs']}
> 2017-06-26 15:30:52,648 - FS Type: 
> 2017-06-26 15:30:52,649 - Directory['/etc/hadoop'] {'mode': 0755}
> 2017-06-26 15:30:52,668 - 
> 

Re: Review Request 60541: Using URI naming format in dfs.datanode.data.dir causing datanodes startup failure

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60541/#review179278
---


Ship it!




Ship It!

- Alejandro Fernandez


On June 29, 2017, 3:18 p.m., Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60541/
> ---
> 
> (Updated June 29, 2017, 3:18 p.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21376
> https://issues.apache.org/jira/browse/AMBARI-21376
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Using URI naming format in dfs.datanode.data.dir causing datanodes startup
> failure.  
> I can replicate the issue locally @  master1.openstacklocal:8080/>  
> See task logs for starting DN for data1.openstacklocal / data2.openstacklocal
> /data3.openstacklocal
> 
> 
> 
> 
> 2017-06-26 15:30:52,442 - Using hadoop conf dir: 
> /usr/hdp/current/hadoop-client/conf
> 2017-06-26 15:30:52,603 - Stack Feature Version Info: stack_version=2.5, 
> version=2.5.3.0-37, current_cluster_version=2.5.3.0-37 -> 2.5.3.0-37
> 2017-06-26 15:30:52,604 - Using hadoop conf dir: 
> /usr/hdp/current/hadoop-client/conf
> User Group mapping (user_group) is missing in the hostLevelParams
> 2017-06-26 15:30:52,605 - Group['livy'] {}
> 2017-06-26 15:30:52,607 - Group['spark'] {}
> 2017-06-26 15:30:52,607 - Group['hadoop'] {}
> 2017-06-26 15:30:52,607 - Group['users'] {}
> 2017-06-26 15:30:52,607 - Group['knox'] {}
> 2017-06-26 15:30:52,608 - User['hive'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,609 - User['zookeeper'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,609 - User['ams'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,610 - User['tez'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2017-06-26 15:30:52,611 - User['livy'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,611 - User['spark'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,612 - User['ambari-qa'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2017-06-26 15:30:52,612 - User['hdfs'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,613 - User['yarn'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,614 - User['mapred'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,614 - User['hbase'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,615 - User['knox'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,616 - User['hcat'] {'gid': 'hadoop', 
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2017-06-26 15:30:52,616 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] 
> {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2017-06-26 15:30:52,618 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh 
> ambari-qa 
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
>  {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
> 2017-06-26 15:30:52,628 - Skipping 
> Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa 
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
>  due to not_if
> 2017-06-26 15:30:52,628 - Directory['/tmp/hbase-hbase'] {'owner': 
> 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
> 2017-06-26 15:30:52,634 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] 
> {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2017-06-26 15:30:52,635 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh 
> hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] 
> {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
> 2017-06-26 15:30:52,645 - Skipping 
> Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase 
> /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due 
> to not_if
> 2017-06-26 15:30:52,646 - Group['hdfs'] {}
> 2017-06-26 15:30:52,647 - User['hdfs'] {'fetch_nonlocal_groups': True, 
> 'groups': ['hadoop', 'hdfs']}
> 2017-06-26 15:30:52,648 - FS Type: 
> 2017-06-26 15:30:52,649 - Directory['/etc/hadoop'] {'mode': 0755}
> 2017-06-26 15:30:52,668 - 
> 

Re: Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/#review179277
---




ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml
Lines 232 (patched)


Same change is needed in trunk for Hive in 3.0


- Alejandro Fernandez


On June 29, 2017, 4:28 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60543/
> ---
> 
> (Updated June 29, 2017, 4:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21364
> https://issues.apache.org/jira/browse/AMBARI-21364
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129
> # Upgrade Ambari to 2.5.2.0-74 (hash: 
> fd30644590991deb41241454d6e9091ed7a38e92)
> # Run "ambari-server start"
> 
> {code}
> root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
> restart
> Using python  /usr/bin/python
> Restarting ambari-server
> Waiting for server stop...
> Ambari Server stopped
> Ambari Server running with administrator privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Ambari database consistency check started...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Waiting for server start..
> DB configs consistency check failed. Run "ambari-server start 
> --skip-database-check" to skip. You may try --auto-fix-database flag to 
> attempt to fix issues automatically. If you use this "--skip-database-check" 
> option, do not make any changes to your cluster topology or perform a cluster 
> upgrade until you correct the database consistency issues. See 
> /var/log/ambari-server/ambari-server-check-database.log for more details on 
> the consistency issues.
> ERROR: Exiting with exit code -1.
> REASON: Ambari Server java process has stopped. Please check the logs for 
> more information.
> {code}
> 
> DB log: ambari-server-check-database.log
> {code}
> 2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
> 2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
> configs that we got from db
> 2017-06-27 13:51:38,748  INFO - Getting services from metainfo
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KNOX
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KERBEROS
> 2017-06-27 13:51:38,749  INFO - Comparing required service configs from stack 
> with mapped service configs from db
> 2017-06-27 13:51:38,751 ERROR - Required config(s): product-info is(are) not 
> available for service SMARTSENSE with service config version 2 in cluster cl1
> 2017-06-27 13:51:38,751 ERROR - Required config(s): parquet-logging is(are) 
> not available for service HIVE with service config version 7 in cluster cl1
> 2017-06-27 13:51:38,751  INFO - Getting services which has mapped configs 
> which are not selected in clusterconfigmapping
> 2017-06-27 13:51:38,755 

Re: Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

[Dmytro Grinenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/#review179275
---


Ship it!




Ship It!

- Dmytro Grinenko


On June 29, 2017, 4:28 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60543/
> ---
> 
> (Updated June 29, 2017, 4:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21364
> https://issues.apache.org/jira/browse/AMBARI-21364
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129
> # Upgrade Ambari to 2.5.2.0-74 (hash: 
> fd30644590991deb41241454d6e9091ed7a38e92)
> # Run "ambari-server start"
> 
> {code}
> root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
> restart
> Using python  /usr/bin/python
> Restarting ambari-server
> Waiting for server stop...
> Ambari Server stopped
> Ambari Server running with administrator privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Ambari database consistency check started...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Waiting for server start..
> DB configs consistency check failed. Run "ambari-server start 
> --skip-database-check" to skip. You may try --auto-fix-database flag to 
> attempt to fix issues automatically. If you use this "--skip-database-check" 
> option, do not make any changes to your cluster topology or perform a cluster 
> upgrade until you correct the database consistency issues. See 
> /var/log/ambari-server/ambari-server-check-database.log for more details on 
> the consistency issues.
> ERROR: Exiting with exit code -1.
> REASON: Ambari Server java process has stopped. Please check the logs for 
> more information.
> {code}
> 
> DB log: ambari-server-check-database.log
> {code}
> 2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
> 2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
> configs that we got from db
> 2017-06-27 13:51:38,748  INFO - Getting services from metainfo
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KNOX
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KERBEROS
> 2017-06-27 13:51:38,749  INFO - Comparing required service configs from stack 
> with mapped service configs from db
> 2017-06-27 13:51:38,751 ERROR - Required config(s): product-info is(are) not 
> available for service SMARTSENSE with service config version 2 in cluster cl1
> 2017-06-27 13:51:38,751 ERROR - Required config(s): parquet-logging is(are) 
> not available for service HIVE with service config version 7 in cluster cl1
> 2017-06-27 13:51:38,751  INFO - Getting services which has mapped configs 
> which are not selected in clusterconfigmapping
> 2017-06-27 13:51:38,755  INFO - Checking Topology tables
> 2017-06-27 13:51:38,757  INFO - Checking for tables with large physical size
> 2017-06-27 13:51:38,759  INFO - The database table host_role_command is 

Re: Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

[Jonathan Hurley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/#review179272
---


Ship it!




Ship It!

- Jonathan Hurley


On June 29, 2017, 12:28 p.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60543/
> ---
> 
> (Updated June 29, 2017, 12:28 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21364
> https://issues.apache.org/jira/browse/AMBARI-21364
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 
> 2.6.1.0-129
> # Upgrade Ambari to 2.5.2.0-74 (hash: 
> fd30644590991deb41241454d6e9091ed7a38e92)
> # Run "ambari-server start"
> 
> {code}
> root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
> restart
> Using python  /usr/bin/python
> Restarting ambari-server
> Waiting for server stop...
> Ambari Server stopped
> Ambari Server running with administrator privileges.
> Organizing resource files at /var/lib/ambari-server/resources...
> Ambari database consistency check started...
> Server PID at: /var/run/ambari-server/ambari-server.pid
> Server out at: /var/log/ambari-server/ambari-server.out
> Server log at: /var/log/ambari-server/ambari-server.log
> Waiting for server start..
> DB configs consistency check failed. Run "ambari-server start 
> --skip-database-check" to skip. You may try --auto-fix-database flag to 
> attempt to fix issues automatically. If you use this "--skip-database-check" 
> option, do not make any changes to your cluster topology or perform a cluster 
> upgrade until you correct the database consistency issues. See 
> /var/log/ambari-server/ambari-server-check-database.log for more details on 
> the consistency issues.
> ERROR: Exiting with exit code -1.
> REASON: Ambari Server java process has stopped. Please check the logs for 
> more information.
> {code}
> 
> DB log: ambari-server-check-database.log
> {code}
> 2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
> 2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
> configs that we got from db
> 2017-06-27 13:51:38,748  INFO - Getting services from metainfo
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
> 2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KNOX
> 2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KERBEROS
> 2017-06-27 13:51:38,749  INFO - Comparing required service configs from stack 
> with mapped service configs from db
> 2017-06-27 13:51:38,751 ERROR - Required config(s): product-info is(are) not 
> available for service SMARTSENSE with service config version 2 in cluster cl1
> 2017-06-27 13:51:38,751 ERROR - Required config(s): parquet-logging is(are) 
> not available for service HIVE with service config version 7 in cluster cl1
> 2017-06-27 13:51:38,751  INFO - Getting services which has mapped configs 
> which are not selected in clusterconfigmapping
> 2017-06-27 13:51:38,755  INFO - Checking Topology tables
> 2017-06-27 13:51:38,757  INFO - Checking for tables with large physical size
> 2017-06-27 13:51:38,759  INFO - The database table host_role_command 

Review Request 60543: DB consistency checker throws errors for missing 'parquet-logging' and 'product-info' configs after Ambari upgrade

[Dmitro Lisnichenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60543/
---

Review request for Ambari, Dmytro Grinenko and Jonathan Hurley.


Bugs: AMBARI-21364
https://issues.apache.org/jira/browse/AMBARI-21364


Repository: ambari


Description
---

*STR*
# Deployed cluster with Ambari version: 2.5.1.0-159 and HDP version: 2.6.1.0-129
# Upgrade Ambari to 2.5.2.0-74 (hash: fd30644590991deb41241454d6e9091ed7a38e92)
# Run "ambari-server start"

{code}
root@ctr-e133-1493418528701-156570-01-05:/hwqe/hadoopqe# ambari-server 
restart
Using python  /usr/bin/python
Restarting ambari-server
Waiting for server stop...
Ambari Server stopped
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start..
DB configs consistency check failed. Run "ambari-server start 
--skip-database-check" to skip. You may try --auto-fix-database flag to attempt 
to fix issues automatically. If you use this "--skip-database-check" option, do 
not make any changes to your cluster topology or perform a cluster upgrade 
until you correct the database consistency issues. See 
/var/log/ambari-server/ambari-server-check-database.log for more details on the 
consistency issues.
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process has stopped. Please check the logs for more 
information.
{code}

DB log: ambari-server-check-database.log
{code}
2017-06-27 13:51:38,743  INFO - Executing query 'GET_SERVICES_WITH_CONFIGS'
2017-06-27 13:51:38,748  INFO - Comparing service configs from stack with 
configs that we got from db
2017-06-27 13:51:38,748  INFO - Getting services from metainfo
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / KAFKA
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / PIG
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / ZEPPELIN
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / LOGSEARCH
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAPREDUCE2
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / SLIDER
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HIVE
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / TEZ
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HBASE
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / OOZIE
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / FLUME
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / MAHOUT
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / HDFS
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / DRUID
2017-06-27 13:51:38,748  INFO - Processing HDP-2.6 / AMBARI_METRICS
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SMARTSENSE
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / AMBARI_INFRA
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / YARN
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / FALCON
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SPARK2
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ZOOKEEPER
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / ATLAS
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / SQOOP
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / STORM
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KNOX
2017-06-27 13:51:38,749  INFO - Processing HDP-2.6 / KERBEROS
2017-06-27 13:51:38,749  INFO - Comparing required service configs from stack 
with mapped service configs from db
2017-06-27 13:51:38,751 ERROR - Required config(s): product-info is(are) not 
available for service SMARTSENSE with service config version 2 in cluster cl1
2017-06-27 13:51:38,751 ERROR - Required config(s): parquet-logging is(are) not 
available for service HIVE with service config version 7 in cluster cl1
2017-06-27 13:51:38,751  INFO - Getting services which has mapped configs which 
are not selected in clusterconfigmapping
2017-06-27 13:51:38,755  INFO - Checking Topology tables
2017-06-27 13:51:38,757  INFO - Checking for tables with large physical size
2017-06-27 13:51:38,759  INFO - The database table host_role_command is 
currently 16.703 MB and is within normal limits (3000.000)
2017-06-27 13:51:38,760  INFO - The database table execution_command is 
currently 12.516 MB and is within normal limits (3000.000)
2017-06-27 13:51:38,761  INFO - The database table stage is currently 0.109 MB 
and is within normal limits (3000.000)
2017-06-27 13:51:38,761  INFO - The database table request is currently 0.078 
MB and is within normal limits (3000.000)
2017-06-27 13:51:38,762  INFO - The database table alert_history is currently 
0.563 MB and is within normal limits (3000.000)

Re: Review Request 60480: AMBARI-21351 Add Support For MariaDB to Python Scripts

[Di Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60480/
---

(Updated June 29, 2017, 3:47 p.m.)


Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan 
Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.


Bugs: AMBARI-21351
https://issues.apache.org/jira/browse/AMBARI-21351


Repository: ambari


Description
---

Some stacks, such as BigInights, support MariaDB as the database for Hive. When 
upgrading from BigInights to another stack, such as HDP, this causes the 
following exception when starting the Hive service:
```resource_management.core.exceptions.ExecutionFailed: Execution of 'service 
mysqld stop' returned 5. Redirecting to /bin/systemctl stop  mysqld.service
Failed to stop mysqld.service: Unit mysqld.service not loaded.```
This looks to be caused by the hard-coded reliance of mysqld in the Python 
scripts.


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py
 b7cb148 


Diff: https://reviews.apache.org/r/60480/diff/4/

Changes: https://reviews.apache.org/r/60480/diff/3-4/


Testing
---

Tim and I patched a cluster with basically the same fix when migrating an IOP 
4.2.5 cluster to HDP 2.6/Ambari 2.5.1. THis patch has one additional change 
that's to set StackFeature boolean in case this code gets merged back to trunk 
later on. 

If the code only stays in this branch, the StackFeature boolean change can be 
omitted.


Thanks,

Di Li

Re: Review Request 60480: AMBARI-21351 Add Support For MariaDB to Python Scripts

[Di Li]

> On June 28, 2017, 7:51 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml
> > Lines 118 (patched)
> > 
> >
> > This is missing 
> 
> Di Li wrote:
> right, I removed it per Jonathan's comment. Or I can add it back but set 
> to false ( not seeing the point of that comparing to not having the line at 
> all)
> 
> " This will cause upgrades to change configs and have Hive report 
> staleness and require a restart. Instead, it seems like we should default to 
> False if not found and use the upgrade packs to update this.
> 
> In other words; since Stack-A might need this when moving to Stack-B, 
> then it should provide this as a configuration setting. "

I think I misundertood Jonathan's comment. It should have been a Py only change 
with default value to false and rely on config update during the migration to 
tell Py to use mariadb instead of mysqld.


- Di


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60480/#review179157
---


On June 28, 2017, 6:23 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60480/
> ---
> 
> (Updated June 28, 2017, 6:23 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21351
> https://issues.apache.org/jira/browse/AMBARI-21351
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Some stacks, such as BigInights, support MariaDB as the database for Hive. 
> When upgrading from BigInights to another stack, such as HDP, this causes the 
> following exception when starting the Hive service:
> ```resource_management.core.exceptions.ExecutionFailed: Execution of 'service 
> mysqld stop' returned 5. Redirecting to /bin/systemctl stop  mysqld.service
> Failed to stop mysqld.service: Unit mysqld.service not loaded.```
> This looks to be caused by the hard-coded reliance of mysqld in the Python 
> scripts.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/configuration/hive-env.xml
>  1cf9016 
>   
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/status_params.py
>  b7cb148 
> 
> 
> Diff: https://reviews.apache.org/r/60480/diff/3/
> 
> 
> Testing
> ---
> 
> Tim and I patched a cluster with basically the same fix when migrating an IOP 
> 4.2.5 cluster to HDP 2.6/Ambari 2.5.1. THis patch has one additional change 
> that's to set StackFeature boolean in case this code gets merged back to 
> trunk later on. 
> 
> If the code only stays in this branch, the StackFeature boolean change can be 
> omitted.
> 
> 
> Thanks,
> 
> Di Li
> 
>

Re: Review Request 60537: AMBARI-21373 Remove clean repo logic not suitable for Apache Ambari Python API

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60537/#review179262
---


Ship it!




Ship It!

- Tim Thorpe


On June 29, 2017, 1:39 p.m., Di Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60537/
> ---
> 
> (Updated June 29, 2017, 1:39 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, 
> Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21373
> https://issues.apache.org/jira/browse/AMBARI-21373
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> The  clean repo logic is internal only, remove it in Apache Ambari as 
> community version does not have necessary Python API in recourse_management. 
> Also the code is not a must-have
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.0/hooks/before-INSTALL/scripts/repo_initialization.py
>  1b0f5c6 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-INSTALL/scripts/repo_initialization.py
>  98fa131 
> 
> 
> Diff: https://reviews.apache.org/r/60537/diff/1/
> 
> 
> Testing
> ---
> 
> Patched an IOP/HDP cluster post Ambari upgrade, before migration.
> 
> 
> Thanks,
> 
> Di Li
> 
>

Review Request 60540: AMBARI-21339 logviewer started along with nimbus if supervisor is not running on the same machine

[Laszlo Puskas]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60540/
---

Review request for Ambari, Attila Doroszlai, Sumit Mohanty, and Sebastian 
Toader.


Bugs: AMBARI-21339
https://issues.apache.org/jira/browse/AMBARI-21339


Repository: ambari


Description
---

Logviewer is only started along with the supervisor so inn cases when nimbus 
runs on a node not hosting supervisor logviever doesn't run.

Solution:
- in such cases the logviewr is started along with nimbus


Diffs
-

  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus.py
 a6bc437 


Diff: https://reviews.apache.org/r/60540/diff/1/


Testing
---

Manually on dev env.

- provision cluster with storm, nimbus and supervisor running on the same host 
-> logviewer started along with *supervisor*
- provision cluster with storm, nimbus running "alone" on the node -> logviewer 
started along with *nimbus*


Thanks,

Laszlo Puskas

Review Request 60541: Using URI naming format in dfs.datanode.data.dir causing datanodes startup failure

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60541/
---

Review request for Ambari and Sumit Mohanty.


Bugs: AMBARI-21376
https://issues.apache.org/jira/browse/AMBARI-21376


Repository: ambari


Description
---

Using URI naming format in dfs.datanode.data.dir causing datanodes startup
failure.  
I can replicate the issue locally @   
See task logs for starting DN for data1.openstacklocal / data2.openstacklocal
/data3.openstacklocal




2017-06-26 15:30:52,442 - Using hadoop conf dir: 
/usr/hdp/current/hadoop-client/conf
2017-06-26 15:30:52,603 - Stack Feature Version Info: stack_version=2.5, 
version=2.5.3.0-37, current_cluster_version=2.5.3.0-37 -> 2.5.3.0-37
2017-06-26 15:30:52,604 - Using hadoop conf dir: 
/usr/hdp/current/hadoop-client/conf
User Group mapping (user_group) is missing in the hostLevelParams
2017-06-26 15:30:52,605 - Group['livy'] {}
2017-06-26 15:30:52,607 - Group['spark'] {}
2017-06-26 15:30:52,607 - Group['hadoop'] {}
2017-06-26 15:30:52,607 - Group['users'] {}
2017-06-26 15:30:52,607 - Group['knox'] {}
2017-06-26 15:30:52,608 - User['hive'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,609 - User['zookeeper'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,609 - User['ams'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,610 - User['tez'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-06-26 15:30:52,611 - User['livy'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,611 - User['spark'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,612 - User['ambari-qa'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-06-26 15:30:52,612 - User['hdfs'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,613 - User['yarn'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,614 - User['mapred'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,614 - User['hbase'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,615 - User['knox'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,616 - User['hcat'] {'gid': 'hadoop', 
'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-06-26 15:30:52,616 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] 
{'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-06-26 15:30:52,618 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh 
ambari-qa 
/tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
 {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-06-26 15:30:52,628 - Skipping 
Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa 
/tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
 due to not_if
2017-06-26 15:30:52,628 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 
'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2017-06-26 15:30:52,634 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] 
{'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-06-26 15:30:52,635 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh 
hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] 
{'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2017-06-26 15:30:52,645 - Skipping 
Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase 
/home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to 
not_if
2017-06-26 15:30:52,646 - Group['hdfs'] {}
2017-06-26 15:30:52,647 - User['hdfs'] {'fetch_nonlocal_groups': True, 
'groups': ['hadoop', 'hdfs']}
2017-06-26 15:30:52,648 - FS Type: 
2017-06-26 15:30:52,649 - Directory['/etc/hadoop'] {'mode': 0755}
2017-06-26 15:30:52,668 - 
File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': 
InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
2017-06-26 15:30:52,669 - 
Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 
'group': 'hadoop', 'mode': 01777}
2017-06-26 15:30:52,687 - Execute[('setenforce', '0')] {'not_if': '(! which 
getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': 
True, 'only_if': 'test -f /selinux/enforce'}
2017-06-26 15:30:52,716 - Directory['/var/log/hadoop'] {'owner': 'root', 
'create_parents': True, 'group': 'hadoop', 'mode': 

Re: Review Request 60514: Use JDK 8 maven compiler plugin for Log Search and Infra projects

[Robert Nettleton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60514/#review179259
---


Ship it!




Ship It!

- Robert Nettleton


On June 28, 2017, 7:34 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60514/
> ---
> 
> (Updated June 28, 2017, 7:34 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Miklos Gergely, Robert 
> Nettleton, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21369
> https://issues.apache.org/jira/browse/AMBARI-21369
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Start to use jdk 8 compiler maven plugin for Log Search and Infra
> 
> 
> Diffs
> -
> 
>   ambari-infra/ambari-infra-manager/pom.xml b7708c2 
>   ambari-infra/ambari-infra-solr-plugin/pom.xml c890cec 
>   ambari-infra/pom.xml 07adb3e 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml ae2150e 
>   ambari-logsearch/ambari-logsearch-server/pom.xml fc4029b 
>   ambari-logsearch/pom.xml af1dc6a 
> 
> 
> Diff: https://reviews.apache.org/r/60514/diff/1/
> 
> 
> Testing
> ---
> 
> done. (fails with jdk 7)
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 60514: Use JDK 8 maven compiler plugin for Log Search and Infra projects

[Attila Doroszlai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60514/#review179251
---


Ship it!




Ship It!

- Attila Doroszlai


On June 28, 2017, 9:34 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60514/
> ---
> 
> (Updated June 28, 2017, 9:34 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Miklos Gergely, Robert 
> Nettleton, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21369
> https://issues.apache.org/jira/browse/AMBARI-21369
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Start to use jdk 8 compiler maven plugin for Log Search and Infra
> 
> 
> Diffs
> -
> 
>   ambari-infra/ambari-infra-manager/pom.xml b7708c2 
>   ambari-infra/ambari-infra-solr-plugin/pom.xml c890cec 
>   ambari-infra/pom.xml 07adb3e 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml ae2150e 
>   ambari-logsearch/ambari-logsearch-server/pom.xml fc4029b 
>   ambari-logsearch/pom.xml af1dc6a 
> 
> 
> Diff: https://reviews.apache.org/r/60514/diff/1/
> 
> 
> Testing
> ---
> 
> done. (fails with jdk 7)
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Review Request 60537: AMBARI-21373 Remove clean repo logic not suitable for Apache Ambari Python API

[Di Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60537/
---

Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan 
Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.


Bugs: AMBARI-21373
https://issues.apache.org/jira/browse/AMBARI-21373


Repository: ambari


Description
---

The  clean repo logic is internal only, remove it in Apache Ambari as community 
version does not have necessary Python API in recourse_management. Also the 
code is not a must-have


Diffs
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.0/hooks/before-INSTALL/scripts/repo_initialization.py
 1b0f5c6 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-INSTALL/scripts/repo_initialization.py
 98fa131 


Diff: https://reviews.apache.org/r/60537/diff/1/


Testing
---

Patched an IOP/HDP cluster post Ambari upgrade, before migration.


Thanks,

Di Li

Review Request 60535: AMBARI-21372. Allow Dependency Checks to be Skipped During Stack Package Installation

[Attila Doroszlai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60535/
---

Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Sumit Mohanty, 
and Sid Wagle.


Bugs: AMBARI-21372
https://issues.apache.org/jira/browse/AMBARI-21372


Repository: ambari


Description
---

Introduce `ignore_dependencies` flag for `ClusterStackVersions` request.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 edf12514a2966d4f29877e93c702f107ed83f37a 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterStackVersionResourceProvider.java
 621d71e9391200be057d1efd6657dd3562d459bb 
  ambari-server/src/main/resources/custom_actions/scripts/install_packages.py 
3a93389957e36686848217fd7f69b8af0101d174 


Diff: https://reviews.apache.org/r/60535/diff/1/


Testing
---

Manual test on CentOS7.

With `ignore_dependencies: true`:

```
$ curl -X POST -d '{ "ClusterStackVersions": { "stack": "HDP", "version": 
"2.6", "repository_version": "2.6.3.0-11", "ignore_dependencies": "true" } }' 
http://localhost:8080/api/v1/clusters/TEST/stack_versions

output-52.txt:
...
2017-06-29 11:57:27,098 - Will install packages for repository version 
2.6.3.0-11
...
2017-06-29 11:57:27,133 - Installing package hdp-select ('/usr/bin/yum -d 0 -e 
0 -y install hdp-select')
...
2017-06-29 11:58:58,669 - Installing package zookeeper_2_6_3_0_11-server 
('/usr/bin/yum -d 0 -e 0 -y install zookeeper_2_6_3_0_11-server')
2017-06-29 11:59:30,007 - Ignoring package dependencies
...
2017-06-29 11:59:30,028 - Configuration symlinks are not needed for 2.6.3.0-11
```

Defaults to `false` if flag is omitted:

```
$ curl -X POST -d '{ "ClusterStackVersions": { "stack": "HDP", "version": 
"2.6", "repository_version": "2.6.3.0-11" } }' 
http://localhost:8080/api/v1/clusters/TEST/stack_versions

output-52.txt:
...
2017-06-29 12:03:02,930 - Will install packages for repository version 
2.6.3.0-11
...
2017-06-29 12:03:02,999 - Installing package hdp-select ('/usr/bin/yum -d 0 -e 
0 -y install hdp-select')
...
2017-06-29 12:04:29,980 - Installing package zookeeper_2_6_3_0_11-server 
('/usr/bin/yum -d 0 -e 0 -y install zookeeper_2_6_3_0_11-server')
2017-06-29 12:05:08,189 - checked_call[['/usr/bin/yum', '-d', '0', '-e', '0', 
'check', 'dependencies']] {'sudo': True}
2017-06-29 12:05:08,953 - Could not install packages. Error: Execution of 
'/usr/bin/yum -d 0 -e 0 check dependencies' returned 1. Error: check 
['dependencies']
...
```


Unit tests (same as on `branch-feature-AMBARI-21348`):

```
Failed tests:
  UpgradeActionTest.testFinalizeDowngrade:515 expected: but 
was:
  UpgradeActionTest.testFinalizeDowngradeAcrossStacks:813 expected:<4> but 
was:<8>
  UpgradeActionTest.testFinalizeUpgrade:606->verifyBaseRepoURL:674 expected:<1> 
but was:<0>
  ConfigUpgradeValidityTest.testValidateConfigUpgradePacks:225 File 
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
 appears to be a config upgrade pack, but does not define 'upgrade-config.xsd' 
as its schema
Tests in error:
  
UpgradeActionTest.testFinalizeDowngradeWhenDidNotFinalizePreviousUpgrade:541->makeTwoUpgradesWhereLastDidNotComplete:254
 » Ambari
  ServicePropertiesTest.validatePropertySchemaOfServiceXMLs:50 » Ambari File 
/Us...

Tests run: 4993, Failures: 4, Errors: 2, Skipped: 39
```


Thanks,

Attila Doroszlai

Re: Review Request 60501: Create property descriptions for internal Log Search configs

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60501/#review179245
---


Ship it!




Ship It!

- Miklos Gergely


On June 28, 2017, 3:51 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60501/
> ---
> 
> (Updated June 28, 2017, 3:51 p.m.)
> 
> 
> Review request for Ambari, Andrii Babiichuk, Miklos Gergely, and Robert 
> Nettleton.
> 
> 
> Bugs: AMBARI-21366
> https://issues.apache.org/jira/browse/AMBARI-21366
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> - added new annotation to create descrptions for all internal logsearch 
> properties (currently only for logsearch.properties, not logfeeder.properties)
> - create /info endpoints to get access the property descriptions 
> (/api/v1/info/properties)
> - speed up doc generation (for this change and for swagger as well...as it 
> generated every time you hit the rest api call, now it will generated during 
> startup on a thread, from that point you will be able to acces the docs from 
> the memory)
> 
> Also some UI changes:
> - there was a /api/v1/pulbic/config call, instead of it, uses 
> /api/v1/info/auth with returning a simple map json (not that weird name value 
> structure)...use that endpoint in login.html
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchPropertyDescription.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/LogSearch.java
>  45281b6 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/common/ApiDocStorage.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/common/PropertiesHelper.java
>  a2a7f5e 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/common/PropertyDescriptionStorage.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/ApiDocConfig.java
>  9d4e96e 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/AuthPropsConfig.java
>  54cc10c 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/LogSearchHttpHeaderConfig.java
>  cb8c097 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SecurityConfig.java
>  441b628 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrAuditLogPropsConfig.java
>  5981bcc 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrConnectionPropsConfig.java
>  7d37efd 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrEventHistoryPropsConfig.java
>  a117501 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrKerberosConfig.java
>  7cf79b0 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrServiceLogPropsConfig.java
>  482438a 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/doc/DocConstants.java
>  73de0ee 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/InfoManager.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/PublicManager.java
>  cbab651 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/response/PropertyDescriptionData.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/rest/InfoResource.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/rest/PublicResource.java
>  df83d44 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/rest/SwaggerResource.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/util/SSLUtil.java
>  d4b6544 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/filters/LogsearchKRBAuthenticationFilter.java
>  1b77753 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/security/LogsearchExternalServerAuthenticationProvider.java
>  1dab126 
>   
> ambari-logsearch/ambari-logsearch-server/src/test/java/org/apache/ambari/logsearch/web/security/LogsearchExternalServerAuthenticationProviderTest.java
>  d6247a1 
>   

Re: Review Request 60201: AMBARI-17382 : Migrate AMS queries to use ROW_TIMESTAMP instead of native timerange hint

[Dmytro Sen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60201/#review179244
---


Ship it!




Ship It!

- Dmytro Sen


On Июнь 21, 2017, 8:39 п.п., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60201/
> ---
> 
> (Updated Июнь 21, 2017, 8:39 п.п.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-17382
> https://issues.apache.org/jira/browse/AMBARI-17382
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> With PHOENIX-914Link, there is a change in implementation , As earlier, 
> timestamp range was passed as a hint to the query to get advantage of native 
> timerange optimization in hbase but with new implementation we can mark the 
> timestamp column in the schema as a ROW_TIMESTAMP and pass timestamp range 
> with “where” clause only to achieve equivalent performance and better 
> accuracy.
> 
> For ex:-
> 
> With earlier implementation , AMS forms query like this:-
> SELECT /*+ NATIVE_TIME_RANGE(1448029523000) */ METRIC_NAME, APP_ID, 
> INSTANCE_ID, SERVER_TIME, UNITS, METRIC_SUM, HOSTS_COUNT, METRIC_MAX, 
> METRIC_MIN FROM METRIC_AGGREGATE WHERE (METRIC_NAME IN 
> ('regionserver.Server.totalRequestCount', 
> 'regionserver.Server.blockCacheCountHitPercent', 
> 'regionserver.Server.regionCount', 
> 'regionserver.Server.compactionQueueLength', 
> 'regionserver.Server.storeFileCount', 'master.Server.averageLoad')) AND 
> APP_ID = 'ams-hbase' AND SERVER_TIME >= 1448029643000 AND SERVER_TIME < 
> 1448033243 ORDER BY METRIC_NAME, SERVER_TIME LIMIT 11520
> 
> 
> WORK DONE
> 1. Removed START_TIME column from AMS schema and access layer.
> 2. Removed timestamp field from TimelineMetric object. 
> 3. From now on, StartTime field in TimelineMetric object maps to 
> 'SERVER_TIME' in metric tables which are of type 'ROW_TIMESTAMP'
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/SingleValuedTimelineMetric.java
>  4bb9355 
>   
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetric.java
>  3d3b19c 
>   
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetrics.java
>  383079a 
>   
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/cache/TimelineMetricsEhCacheSizeOfEngine.java
>  0e23e17 
>   
> ambari-metrics/ambari-metrics-hadoop-sink/src/main/java/org/apache/hadoop/metrics2/sink/timeline/HadoopTimelineMetricsSink.java
>  a290ced 
>   ambari-metrics/ambari-metrics-timelineservice/pom.xml 34794e6 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
>  7ad88a1 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStoreWatcher.java
>  aa53430 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  83f2392 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregator.java
>  0f6dd79 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java
>  cae7263 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricHostAggregator.java
>  8f941e1 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
>  c8b5728 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricClusterAggregator.java
>  c7b605f 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/v2/TimelineMetricHostAggregator.java
>  57a3034 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/query/PhoenixTransactSQL.java
>  51c96c6 
>   
> 

Re: Review Request 60514: Use JDK 8 maven compiler plugin for Log Search and Infra projects

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60514/#review179243
---


Ship it!




Ship It!

- Miklos Gergely


On June 28, 2017, 7:34 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60514/
> ---
> 
> (Updated June 28, 2017, 7:34 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Miklos Gergely, Robert 
> Nettleton, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21369
> https://issues.apache.org/jira/browse/AMBARI-21369
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Start to use jdk 8 compiler maven plugin for Log Search and Infra
> 
> 
> Diffs
> -
> 
>   ambari-infra/ambari-infra-manager/pom.xml b7708c2 
>   ambari-infra/ambari-infra-solr-plugin/pom.xml c890cec 
>   ambari-infra/pom.xml 07adb3e 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml ae2150e 
>   ambari-logsearch/ambari-logsearch-server/pom.xml fc4029b 
>   ambari-logsearch/pom.xml af1dc6a 
> 
> 
> Diff: https://reviews.apache.org/r/60514/diff/1/
> 
> 
> Testing
> ---
> 
> done. (fails with jdk 7)
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>