date:20171213

Re: Review Request 64544: Migrate user data for upgrade to improved user account management

2017-12-13 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64544/
---

(Updated Dec. 13, 2017, 5:01 p.m.)


Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene Chekanskiy, 
Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Molnar.


Changes
---

Fixed DB scheama concerns.


Bugs: AMBARI-22577
https://issues.apache.org/jira/browse/AMBARI-22577


Repository: ambari


Description
---

Migrate data from the users table (pre-Ambari 3.0.0) to the updated users table 
and user_authentication tables.


Diffs (updated)
-

  ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessor.java 
549c0fd7e8 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/UserAuthenticationDAO.java
 513e78200d 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UserAuthenticationEntity.java
 27514f648c 
  
ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProvider.java
 20a06ccd54 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
 2de60957bc 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e58a04e4b2 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql cc589e47f2 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 7cd083db4c 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql ff232ebe3e 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql d7c09f3381 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b4929b308d 
  
ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
 29f9d917e8 
  
ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderForDNWithSpaceTest.java
 10076b0876 
  
ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderTest.java
 43d4d6b0c0 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog300Test.java
 747f99b618 


Diff: https://reviews.apache.org/r/64544/diff/2/

Changes: https://reviews.apache.org/r/64544/diff/1-2/


Testing
---

manually tested

# Local test results: 
```
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 27:21 min
[INFO] Finished at: 2017-12-11T17:30:31-05:00
[INFO] Final Memory: 100M/2019M
[INFO] 
```

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 64553: Log Feeder refactor: integrate with spring boot

2017-12-13 Thread Oliver Szabo



> On Dec. 13, 2017, 9:18 p.m., Sid Wagle wrote:
> > ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
> > Lines 40 (patched)
> > 
> >
> > When would this be false?

in the ApplicationConfig, then the object will be ready during setting up 
application context, and an other one can start the iniialization


> On Dec. 13, 2017, 9:18 p.m., Sid Wagle wrote:
> > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/StatsLogger.java
> > Lines 30 (patched)
> > 
> >
> > More contemporary design would be to create a Runnable that can be used 
> > by the caller as an argument scheduledExectuor. The scehduling logic can be 
> > kept out of the class.

as i mentioned Krisztina that patch focus on spring boot support so that class 
was just moved here, probably we will get rid of it


- Oliver


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64553/#review193736
---


On Dec. 13, 2017, 3:24 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64553/
> ---
> 
> (Updated Dec. 13, 2017, 3:24 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa and Sid Wagle.
> 
> 
> Bugs: AMBARI-22639
> https://issues.apache.org/jira/browse/AMBARI-22639
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> First patch for refactor logfeeder - Spring Boot support
> 
> Add spring boot support, the goal is to get rid of LogFeederPropertiesUtil 
> and let Spring to handle configs & beans.
> - remove LogFeederPropertiesUtil
> - move logfeeder properties fields into mutliple POJOs
> - update some tests & remove one test (needs to rewrite it)
> - convert singleton classes to spring beans
> - let Spring to handle application pid file
> - convert some internal threads to classes (like StatsLogger)
> 
> (it does not contain any real changes regarding to input-filter-output blocks)
> 
> Next patch will be focus on the usage of start scritps (in ambari-logsearch 
> and ambari-server stack code)
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
>  a84a97b 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java
>  6ed36fd 
>   ambari-logsearch/ambari-logsearch-logfeeder/.gitignore PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 01710bf 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  5114743 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeederCommandLine.java
>  d996f98 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
>  243b344 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigItem.java
>  5c20a8e 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
>  ec29f69 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
>  a76 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/ApplicationConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/InputSimulateConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogEntryCacheConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/MetricsCollectorConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/Filter.java
>  8e8834b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java
>  fc7a565 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterKeyValue.java
>  8e5aee8 
>   
> ambari-logse

Re: Review Request 64553: Log Feeder refactor: integrate with spring boot

2017-12-13 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64553/#review193736
---


Ship it!





ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
Lines 40 (patched)


When would this be false?



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/StatsLogger.java
Lines 30 (patched)


More contemporary design would be to create a Runnable that can be used by 
the caller as an argument scheduledExectuor. The scehduling logic can be kept 
out of the class.


- Sid Wagle


On Dec. 13, 2017, 3:24 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64553/
> ---
> 
> (Updated Dec. 13, 2017, 3:24 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa and Sid Wagle.
> 
> 
> Bugs: AMBARI-22639
> https://issues.apache.org/jira/browse/AMBARI-22639
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> First patch for refactor logfeeder - Spring Boot support
> 
> Add spring boot support, the goal is to get rid of LogFeederPropertiesUtil 
> and let Spring to handle configs & beans.
> - remove LogFeederPropertiesUtil
> - move logfeeder properties fields into mutliple POJOs
> - update some tests & remove one test (needs to rewrite it)
> - convert singleton classes to spring beans
> - let Spring to handle application pid file
> - convert some internal threads to classes (like StatsLogger)
> 
> (it does not contain any real changes regarding to input-filter-output blocks)
> 
> Next patch will be focus on the usage of start scritps (in ambari-logsearch 
> and ambari-server stack code)
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
>  a84a97b 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java
>  6ed36fd 
>   ambari-logsearch/ambari-logsearch-logfeeder/.gitignore PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 01710bf 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  5114743 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeederCommandLine.java
>  d996f98 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
>  243b344 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigItem.java
>  5c20a8e 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
>  ec29f69 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
>  a76 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/ApplicationConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/InputSimulateConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogEntryCacheConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/MetricsCollectorConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/Filter.java
>  8e8834b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java
>  fc7a565 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterKeyValue.java
>  8e5aee8 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
>  b021c37 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
>  972011d 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
>  8f8c4fd 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache

Re: Review Request 64537: AMBARI-22635 : Ambari should create a dummy core-site.xml for Ranger plugins when namenode is not installed.

2017-12-13 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64537/#review193734
---


Ship it!




Ship It!

- Sid Wagle


On Dec. 13, 2017, 3:42 p.m., Vishal Suvagia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64537/
> ---
> 
> (Updated Dec. 13, 2017, 3:42 p.m.)
> 
> 
> Review request for Ambari, Gautam Borad, Jonathan Hurley, Jayush Luniya, 
> Madhuvanthi Radhakrishnan, Mugdha Varadkar, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-22635
> https://issues.apache.org/jira/browse/AMBARI-22635
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> For Ranger plugins to work properly in kerberised environments where HDFS is 
> not installed. We need to create a core-site.xml for Storm and Kafka plugins 
> so that the plugins can work to fetch latest policies from with kerberised 
> calls from Ranger.
> Attached patch is for trunk branch, seperate patch for branch-2.6 is attached 
> to Apache JIRA.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
>  485c1a6 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/setup_ranger_kafka.py
>  e9719aa 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/setup_ranger_kafka.py
>  9aa09df 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/setup_ranger_knox.py
>  c486ef7 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/setup_ranger_knox.py
>  c486ef7 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/setup_ranger_storm.py
>  c04496e 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/package/scripts/setup_ranger_storm.py
>  c04496e 
> 
> 
> Diff: https://reviews.apache.org/r/64537/diff/2/
> 
> 
> Testing
> ---
> 
> Verified with installation on Cent-OS 6.
> 
> 
> Thanks,
> 
> Vishal Suvagia
> 
>

Re: Review Request 64537: AMBARI-22635 : Ambari should create a dummy core-site.xml for Ranger plugins when namenode is not installed.

2017-12-13 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64537/#review193731
---


Ship it!




Ship It!

- Jonathan Hurley


On Dec. 13, 2017, 10:42 a.m., Vishal Suvagia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64537/
> ---
> 
> (Updated Dec. 13, 2017, 10:42 a.m.)
> 
> 
> Review request for Ambari, Gautam Borad, Jonathan Hurley, Jayush Luniya, 
> Madhuvanthi Radhakrishnan, Mugdha Varadkar, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-22635
> https://issues.apache.org/jira/browse/AMBARI-22635
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> For Ranger plugins to work properly in kerberised environments where HDFS is 
> not installed. We need to create a core-site.xml for Storm and Kafka plugins 
> so that the plugins can work to fetch latest policies from with kerberised 
> calls from Ranger.
> Attached patch is for trunk branch, seperate patch for branch-2.6 is attached 
> to Apache JIRA.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
>  485c1a6 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/setup_ranger_kafka.py
>  e9719aa 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/setup_ranger_kafka.py
>  9aa09df 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/setup_ranger_knox.py
>  c486ef7 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/setup_ranger_knox.py
>  c486ef7 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/setup_ranger_storm.py
>  c04496e 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/package/scripts/setup_ranger_storm.py
>  c04496e 
> 
> 
> Diff: https://reviews.apache.org/r/64537/diff/2/
> 
> 
> Testing
> ---
> 
> Verified with installation on Cent-OS 6.
> 
> 
> Thanks,
> 
> Vishal Suvagia
> 
>

Re: Review Request 64544: Migrate user data for upgrade to improved user account management

2017-12-13 Thread Robert Levas



> On Dec. 12, 2017, 11:44 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql
> > Lines 291 (patched)
> > 
> >
> > You should create the FK declarations inside the table declarations.
> 
> Robert Levas wrote:
> I did this for a reason, but now I can't remember. I will test it and fix 
> or post my reason.

Looks like I was doing it for consistency... The IntelliJ Oracle SQL doc 
validator does not seem to like it when the CREATE TABLE statement contains an 
INDEX statement. So maybe creating indicies in Oracle needs to be done external 
to the CREATE TABLE statement.  I am not sure how bad it is, but I removed the 
index that I was creating and moved the FK constrains into the CREATE TABLE 
statments.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64544/#review193564
---


On Dec. 12, 2017, 9:35 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64544/
> ---
> 
> (Updated Dec. 12, 2017, 9:35 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene 
> Chekanskiy, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Molnar.
> 
> 
> Bugs: AMBARI-22577
> https://issues.apache.org/jira/browse/AMBARI-22577
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Migrate data from the users table (pre-Ambari 3.0.0) to the updated users 
> table and user_authentication tables.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessor.java 
> 549c0fd7e8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/UserAuthenticationDAO.java
>  513e78200d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UserAuthenticationEntity.java
>  27514f648c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProvider.java
>  20a06ccd54 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
>  2de60957bc 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e58a04e4b2 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql cc589e47f2 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 7cd083db4c 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql ff232ebe3e 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 
> d7c09f3381 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b4929b308d 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
>  29f9d917e8 
>   ambari-server/src/test/java/org/apache/ambari/server/orm/db/DDLTests.java 
> 96cf64e53c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderForDNWithSpaceTest.java
>  10076b0876 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderTest.java
>  43d4d6b0c0 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog300Test.java
>  747f99b618 
> 
> 
> Diff: https://reviews.apache.org/r/64544/diff/1/
> 
> 
> Testing
> ---
> 
> manually tested
> 
> # Local test results: 
> ```
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 27:21 min
> [INFO] Finished at: 2017-12-11T17:30:31-05:00
> [INFO] Final Memory: 100M/2019M
> [INFO] 
> 
> ```
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 64544: Migrate user data for upgrade to improved user account management

2017-12-13 Thread Robert Levas



> On Dec. 12, 2017, 11:44 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
> > Lines 175-192 (patched)
> > 
> >
> > Interesting .. is this idempotent? Typically nested-selects with 
> > inserts are not if they could violate PKs. Maybe you need to breakt this 
> > out into 2 queries and ensure you're only inserting when necessary?
> > 
> > Then again, maybe I don't quite see what this query is doing and it's 
> > ok... Thought I'd flag it for discussion.

I assumed that this would fail if run multiple times. This is why there is a 
conditional right before it:
```
// Migrate data from users table (if not yet upgraded)
if (!usersTableUpgraded()) {
  dbAccessor.executeUpdate(
  "insert into " + USER_AUTHENTICATION_TABLE +
  "(" + USER_AUTHENTICATION_USER_AUTHENTICATION_ID_COLUMN + ", " + 
USER_AUTHENTICATION_USER_ID_COLUMN + ", " + 
USER_AUTHENTICATION_AUTHENTICATION_TYPE_COLUMN + ", " + 
USER_AUTHENTICATION_AUTHENTICATION_KEY_COLUMN + ", " + 
USER_AUTHENTICATION_CREATE_TIME_COLUMN + ", " + 
USER_AUTHENTICATION_UPDATE_TIME_COLUMN + ")" +
   ...
  );
}
```

Breaking this apart into several queries and a set of insert statements might 
help; but I thought that it would be difficult since the updated UserEntity 
class would be out of sync with the original table. What do others do in this 
situation?  Maybe if I need to go this route, I would create a new `user` table 
rather than upgrade the `users` table?  In the end, I would drop the old `user` 
table, but the OldUserEntitiy class will need to live for many versions of 
Ambari.


> On Dec. 12, 2017, 11:44 a.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql
> > Lines 291 (patched)
> > 
> >
> > You should create the FK declarations inside the table declarations.

I did this for a reason, but now I can't remember. I will test it and fix or 
post my reason.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64544/#review193564
---


On Dec. 12, 2017, 9:35 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64544/
> ---
> 
> (Updated Dec. 12, 2017, 9:35 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene 
> Chekanskiy, Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Molnar.
> 
> 
> Bugs: AMBARI-22577
> https://issues.apache.org/jira/browse/AMBARI-22577
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Migrate data from the users table (pre-Ambari 3.0.0) to the updated users 
> table and user_authentication tables.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessor.java 
> 549c0fd7e8 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/UserAuthenticationDAO.java
>  513e78200d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UserAuthenticationEntity.java
>  27514f648c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProvider.java
>  20a06ccd54 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
>  2de60957bc 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e58a04e4b2 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql cc589e47f2 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 7cd083db4c 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql ff232ebe3e 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 
> d7c09f3381 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql b4929b308d 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
>  29f9d917e8 
>   ambari-server/src/test/java/org/apache/ambari/server/orm/db/DDLTests.java 
> 96cf64e53c 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderForDNWithSpaceTest.java
>  10076b0876 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProviderTest.java
>  43d4d6b0c0 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog300Test.java
>  747f99b618 
> 
> 
> Diff: https://reviews.apache.org/r/64544/diff/1/
> 
> 
> Testing
> ---
> 
> manually tested
> 
> # Local test results: 
> ```
> [INFO] 
> --

Re: Review Request 64579: Node Managers fail to start after Spark2 is patched due to CNF YarnShuffleService

2017-12-13 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64579/#review193721
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 13, 2017, 5:12 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64579/
> ---
> 
> (Updated Dec. 13, 2017, 5:12 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22644
> https://issues.apache.org/jira/browse/AMBARI-22644
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deploy HDP-2.6.4.0 cluster with Ambari-2.6.1.0-114
> # Apply HBase patch Upgrade on the cluster (this step is optional)
> # Then apply Spark2 patch Upgrade on the cluster
> # Restart Node Managers
> 
> *Result*
> NM restart fails with below error:
> ```
> 2017-12-10 07:17:02,559 INFO  impl.MetricsSystemImpl 
> (MetricsSystemImpl.java:shutdown(606)) - NodeManager metrics system shutdown 
> complete.
> 2017-12-10 07:17:02,559 FATAL nodemanager.NodeManager 
> (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:197)
> at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:131)
> at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 8 more
> 2017-12-10 07:17:02,562 INFO  nodemanager.NodeManager 
> (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
> ```
> 
> The spark properties are correctly being written out as per AMBARI-22525.
> 
> Initially, we had defined Spark properties for ATS like this:
> ```xml
> yarn.nodemanager.aux-services.spark_shuffle.classpath
> {{stack_root}}/${hdp.version}/spark/aux/*
> ```
> 
> When YARN upgrades without Spark, we run into AMBARI-22525. Seems like the 
> shuffle classes are installed as part of RPM dependencies, but not the 
> SparkATSPlugin.
> 
> So:
> - If we use YARN's version for the Spark classes, then ATS can't find 
> SparkATSPlugin since that is not part of YARN.
> - If we use Spark's version for the classes, then Spark can never upgrade 
> without YARN since NodeManager can't find the new Spark classes. 
> 
> However, it seems like shuffle and ATS use different properties. We changed 
> all 3 properties in AMBARI-22525:
> 
> ```
> yarn.nodemanager.aux-services.spark2_shuffle.classpath
> yarn.nodemanager.aux-services.spark_shuffle.classpath
> yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
> ```
> 
> It seems like what need to do is change the spark shuffle stuff back to 
> hdp.version, but leave ATS using the new version since we're guaranteed to 
> have Spark installed on the ATS machine.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  6b5559cf91 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/yarn-site.xml
>  29833fbe03 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> ea0e2cd46b 
> 
> 
> Diff: https://reviews.apache.org/r/64579/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing on a patched cluster with YARN/Spark
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64554: HBase Cannot Find LZO Classes After Being Patched

2017-12-13 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64554/#review193720
---


Ship it!




Ship It!

- Dmytro Grinenko


On Dec. 13, 2017, 3:46 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64554/
> ---
> 
> (Updated Dec. 13, 2017, 3:46 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22640
> https://issues.apache.org/jira/browse/AMBARI-22640
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After patching HBase where LZO compression is being used, the following is 
> seen:
> 
> ```
> 2017-12-10 22:31:09,244|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exception in thread "main" 
> java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> com.hadoop.compression.lzo.LzoCodec
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:130)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:301)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.2017-12-10
>  22:31:09,267|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.lang.Thread.run(Thread.java:745)
> 2017-12-10 22:31:09,741|INFO|MainThread|machine.py:189 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exit Code: 1
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
>  ac71ce4b36 
> 
> 
> Diff: https://reviews.apache.org/r/64554/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing on Hbase cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64118: Refactor internal code of handling info between kerberos wizard actions

2017-12-13 Thread Eugene Chekanskiy



> On Dec. 12, 2017, 6:54 p.m., Robert Levas wrote:
> > This is looking really good.  Upgrade logic to the appropriate 
> > UpgradeCatalog class.

Upgrade will be seperate patch. I want to finish server and agent side changes 
to implement upgrade after all tables and data are stable.


> On Dec. 12, 2017, 6:54 p.m., Robert Levas wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
> > Lines 456-464 (patched)
> > 
> >
> > Did you intend to leave this here, or will it be fixed before this is 
> > committed?

Most likely it will be removed after agent side refactoring. On final stage of 
refactoring all TODOs will be reviewed and removed.


- Eugene


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64118/#review193576
---


On Dec. 13, 2017, 6:14 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64118/
> ---
> 
> (Updated Dec. 13, 2017, 6:14 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, and Robert Levas.
> 
> 
> Bugs: AMBARI-22530
> https://issues.apache.org/jira/browse/AMBARI-22530
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Moved out of data files for principal generation.
> Now before every kerberos action all info written to database.
> 
> Server actions determine what principals to process based on filters passed 
> to action, this allow:
> 
> * selective generation
> * split generation process per-host/per-service/per-component basis and make 
> generation concurrent for seperate parts
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  53cceb0ded 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  83d2c9808d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java
>  a7b9d80df0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
>  749943dc00 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  ab85aa1d7c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostKerberosIdentityResourceProvider.java
>  52ab9b56ae 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabDAO.java
>  a8723b7bfa 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabPrincipalDAO.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalDAO.java
>  81e4b3d0bf 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalHostDAO.java
>  f27dc48ea4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostGroupComponentEntityPK.java
>  0898133bb5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabEntity.java
>  a25931b946 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabPrincipalEntity.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabServiceMappingEntity.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalEntity.java
>  5dd54ca30c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntity.java
>  d4e80c65d2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntityPK.java
>  7e57e4ae95 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  b8affb4e19 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CleanupServerAction.java
>  002076d85c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/ConfigureAmbariIdentitiesServerAction.java
>  338415280f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java
>  5ec4c1011e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreatePrincipalsServerAction.java
>  0c906592af 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/DestroyPrincipalsServerAction.java
>  4c80bd425e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/FinalizeKerberosServerAction.java
>  bfd5e4036d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosServerAction.java
>  ff5f5cef1c 
>   
> ambari-serv

Re: Review Request 64118: Refactor internal code of handling info between kerberos wizard actions

2017-12-13 Thread Eugene Chekanskiy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64118/
---

(Updated Dec. 13, 2017, 6:14 p.m.)


Review request for Ambari, Attila Magyar, Balázs Bence Sári, and Robert Levas.


Bugs: AMBARI-22530
https://issues.apache.org/jira/browse/AMBARI-22530


Repository: ambari


Description
---

Moved out of data files for principal generation.
Now before every kerberos action all info written to database.

Server actions determine what principals to process based on filters passed to 
action, this allow:

* selective generation
* split generation process per-host/per-service/per-component basis and make 
generation concurrent for seperate parts


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 53cceb0ded 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
 83d2c9808d 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java
 a7b9d80df0 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
 749943dc00 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 ab85aa1d7c 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostKerberosIdentityResourceProvider.java
 52ab9b56ae 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabDAO.java
 a8723b7bfa 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosKeytabPrincipalDAO.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalDAO.java
 81e4b3d0bf 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/KerberosPrincipalHostDAO.java
 f27dc48ea4 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/HostGroupComponentEntityPK.java
 0898133bb5 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabEntity.java
 a25931b946 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabPrincipalEntity.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosKeytabServiceMappingEntity.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalEntity.java
 5dd54ca30c 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntity.java
 d4e80c65d2 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/KerberosPrincipalHostEntityPK.java
 7e57e4ae95 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
 b8affb4e19 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CleanupServerAction.java
 002076d85c 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/ConfigureAmbariIdentitiesServerAction.java
 338415280f 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreateKeytabFilesServerAction.java
 5ec4c1011e 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/CreatePrincipalsServerAction.java
 0c906592af 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/DestroyPrincipalsServerAction.java
 4c80bd425e 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/FinalizeKerberosServerAction.java
 bfd5e4036d 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosServerAction.java
 ff5f5cef1c 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareEnableKerberosServerAction.java
 671ad95c8f 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareKerberosIdentitiesServerAction.java
 83a2106afd 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/KerberosKeytabController.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/ResolvedKerberosKeytab.java
 17e484ad10 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/stageutils/ResolvedKerberosPrincipal.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/PreconfigureKerberosAction.java
 ca78dbb8a2 
  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
 5ac1ac316d 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql 7045240b30 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql c950c7ef83 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql 537ae196c5 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql b4952c2e86 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 4fb0d0981a 
  ambari-server/src/mai

Re: Review Request 64568: AMBARI-22638. yarn.timeline-service property enabled overriding preset secure blueprint config for the same

2017-12-13 Thread Robert Nettleton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64568/#review193711
---


Ship it!




Ship It!

- Robert Nettleton


On Dec. 13, 2017, 6:54 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64568/
> ---
> 
> (Updated Dec. 13, 2017, 6:54 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Robert Levas, 
> and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22638
> https://issues.apache.org/jira/browse/AMBARI-22638
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Changed `yarn.timeline-service.enabled` in `kerberos.json` to `false`, to 
> match stack default for HDP 3.0.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/kerberos.json 
> a3cfaea95918e629f02d2fa9d6eb72a5c77e5987 
> 
> 
> Diff: https://reviews.apache.org/r/64568/diff/1/
> 
> 
> Testing
> ---
> 
> Tested secure cluster deployment with HDP 3.0, verified value of 
> `yarn.timeline-service.enabled` property: `false` if unset or set to `false` 
> in blueprint, `true` if set to `true` in blueprint.
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-server clean test -Dtest='kerberos/*'
> ...
> Tests run: 205, Failures: 0, Errors: 0, Skipped: 1
> ...
> Total run:1210
> Total errors:0
> Total failures:0
> ...
> BUILD SUCCESS
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 64568: AMBARI-22638. yarn.timeline-service property enabled overriding preset secure blueprint config for the same

2017-12-13 Thread Balázs Bence Sári


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64568/#review193710
---


Ship it!




Ship It!

- Balázs Bence Sári


On Dec. 13, 2017, 6:54 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64568/
> ---
> 
> (Updated Dec. 13, 2017, 6:54 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Robert Levas, 
> and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22638
> https://issues.apache.org/jira/browse/AMBARI-22638
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Changed `yarn.timeline-service.enabled` in `kerberos.json` to `false`, to 
> match stack default for HDP 3.0.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/kerberos.json 
> a3cfaea95918e629f02d2fa9d6eb72a5c77e5987 
> 
> 
> Diff: https://reviews.apache.org/r/64568/diff/1/
> 
> 
> Testing
> ---
> 
> Tested secure cluster deployment with HDP 3.0, verified value of 
> `yarn.timeline-service.enabled` property: `false` if unset or set to `false` 
> in blueprint, `true` if set to `true` in blueprint.
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-server clean test -Dtest='kerberos/*'
> ...
> Tests run: 205, Failures: 0, Errors: 0, Skipped: 1
> ...
> Total run:1210
> Total errors:0
> Total failures:0
> ...
> BUILD SUCCESS
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 64579: Node Managers fail to start after Spark2 is patched due to CNF YarnShuffleService

2017-12-13 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64579/#review193706
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On Dec. 13, 2017, 7:12 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64579/
> ---
> 
> (Updated Dec. 13, 2017, 7:12 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22644
> https://issues.apache.org/jira/browse/AMBARI-22644
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deploy HDP-2.6.4.0 cluster with Ambari-2.6.1.0-114
> # Apply HBase patch Upgrade on the cluster (this step is optional)
> # Then apply Spark2 patch Upgrade on the cluster
> # Restart Node Managers
> 
> *Result*
> NM restart fails with below error:
> ```
> 2017-12-10 07:17:02,559 INFO  impl.MetricsSystemImpl 
> (MetricsSystemImpl.java:shutdown(606)) - NodeManager metrics system shutdown 
> complete.
> 2017-12-10 07:17:02,559 FATAL nodemanager.NodeManager 
> (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:197)
> at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:131)
> at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 8 more
> 2017-12-10 07:17:02,562 INFO  nodemanager.NodeManager 
> (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
> ```
> 
> The spark properties are correctly being written out as per AMBARI-22525.
> 
> Initially, we had defined Spark properties for ATS like this:
> ```xml
> yarn.nodemanager.aux-services.spark_shuffle.classpath
> {{stack_root}}/${hdp.version}/spark/aux/*
> ```
> 
> When YARN upgrades without Spark, we run into AMBARI-22525. Seems like the 
> shuffle classes are installed as part of RPM dependencies, but not the 
> SparkATSPlugin.
> 
> So:
> - If we use YARN's version for the Spark classes, then ATS can't find 
> SparkATSPlugin since that is not part of YARN.
> - If we use Spark's version for the classes, then Spark can never upgrade 
> without YARN since NodeManager can't find the new Spark classes. 
> 
> However, it seems like shuffle and ATS use different properties. We changed 
> all 3 properties in AMBARI-22525:
> 
> ```
> yarn.nodemanager.aux-services.spark2_shuffle.classpath
> yarn.nodemanager.aux-services.spark_shuffle.classpath
> yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
> ```
> 
> It seems like what need to do is change the spark shuffle stuff back to 
> hdp.version, but leave ATS using the new version since we're guaranteed to 
> have Spark installed on the ATS machine.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  6b5559cf91 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/yarn-site.xml
>  29833fbe03 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> ea0e2cd46b 
> 
> 
> Diff: https://reviews.apache.org/r/64579/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing on a patched cluster with YARN/Spark
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64542: Kerberos support for OneFS

2017-12-13 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64542/#review193704
---


Ship it!




Ship It!

- Robert Levas


On Dec. 12, 2017, 10:38 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64542/
> ---
> 
> (Updated Dec. 12, 2017, 10:38 a.m.)
> 
> 
> Review request for Ambari, Eugene Chekanskiy, Nate Cole, Robert Levas, and 
> Sandor Molnar.
> 
> 
> Bugs: AMBARI-22634
> https://issues.apache.org/jira/browse/AMBARI-22634
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When kerberos is enabled, the OneFS service check should do a kinit before 
> executing hdfs commands. I also removed some unused code from the 
> servicecheck script.
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/main/admin/kerberos/step1_controller.js 
> 9c864a8f48889515d5b72c0feaf340b7b9e1fe31 
>   ambari-web/app/messages.js b3051962f56951fc6bae169b63c5f029c1ef301d 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/kerberos.json
>  PRE-CREATION 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/package/scripts/params_linux.py
>  b823fc54b84e1149164e17504046a7a32070df1c 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/package/scripts/service_check.py
>  3d798a3342ced3d99a4a2c955a6aca27108c 
> 
> 
> Diff: https://reviews.apache.org/r/64542/diff/1/
> 
> 
> Testing
> ---
> 
> - enabled kerberos, ran service checks on onefs
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 64554: HBase Cannot Find LZO Classes After Being Patched

2017-12-13 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64554/#review193705
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On Dec. 13, 2017, 5:46 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64554/
> ---
> 
> (Updated Dec. 13, 2017, 5:46 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22640
> https://issues.apache.org/jira/browse/AMBARI-22640
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After patching HBase where LZO compression is being used, the following is 
> seen:
> 
> ```
> 2017-12-10 22:31:09,244|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exception in thread "main" 
> java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> com.hadoop.compression.lzo.LzoCodec
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:130)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:301)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.2017-12-10
>  22:31:09,267|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.lang.Thread.run(Thread.java:745)
> 2017-12-10 22:31:09,741|INFO|MainThread|machine.py:189 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exit Code: 1
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
>  ac71ce4b36 
> 
> 
> Diff: https://reviews.apache.org/r/64554/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing on Hbase cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64579: Node Managers fail to start after Spark2 is patched due to CNF YarnShuffleService

2017-12-13 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64579/#review193703
---


Ship it!




Ship It!

- Nate Cole


On Dec. 13, 2017, 12:12 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64579/
> ---
> 
> (Updated Dec. 13, 2017, 12:12 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22644
> https://issues.apache.org/jira/browse/AMBARI-22644
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *STR*
> # Deploy HDP-2.6.4.0 cluster with Ambari-2.6.1.0-114
> # Apply HBase patch Upgrade on the cluster (this step is optional)
> # Then apply Spark2 patch Upgrade on the cluster
> # Restart Node Managers
> 
> *Result*
> NM restart fails with below error:
> ```
> 2017-12-10 07:17:02,559 INFO  impl.MetricsSystemImpl 
> (MetricsSystemImpl.java:shutdown(606)) - NodeManager metrics system shutdown 
> complete.
> 2017-12-10 07:17:02,559 FATAL nodemanager.NodeManager 
> (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.spark.network.yarn.YarnShuffleService
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:197)
> at 
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:131)
> at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 8 more
> 2017-12-10 07:17:02,562 INFO  nodemanager.NodeManager 
> (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
> ```
> 
> The spark properties are correctly being written out as per AMBARI-22525.
> 
> Initially, we had defined Spark properties for ATS like this:
> ```xml
> yarn.nodemanager.aux-services.spark_shuffle.classpath
> {{stack_root}}/${hdp.version}/spark/aux/*
> ```
> 
> When YARN upgrades without Spark, we run into AMBARI-22525. Seems like the 
> shuffle classes are installed as part of RPM dependencies, but not the 
> SparkATSPlugin.
> 
> So:
> - If we use YARN's version for the Spark classes, then ATS can't find 
> SparkATSPlugin since that is not part of YARN.
> - If we use Spark's version for the classes, then Spark can never upgrade 
> without YARN since NodeManager can't find the new Spark classes. 
> 
> However, it seems like shuffle and ATS use different properties. We changed 
> all 3 properties in AMBARI-22525:
> 
> ```
> yarn.nodemanager.aux-services.spark2_shuffle.classpath
> yarn.nodemanager.aux-services.spark_shuffle.classpath
> yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
> ```
> 
> It seems like what need to do is change the spark shuffle stuff back to 
> hdp.version, but leave ATS using the new version since we're guaranteed to 
> have Spark installed on the ATS machine.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
>  6b5559cf91 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/yarn-site.xml
>  29833fbe03 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> ea0e2cd46b 
> 
> 
> Diff: https://reviews.apache.org/r/64579/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing on a patched cluster with YARN/Spark
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64554: HBase Cannot Find LZO Classes After Being Patched

2017-12-13 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64554/#review193702
---


Ship it!




Ship It!

- Nate Cole


On Dec. 13, 2017, 10:46 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64554/
> ---
> 
> (Updated Dec. 13, 2017, 10:46 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22640
> https://issues.apache.org/jira/browse/AMBARI-22640
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After patching HBase where LZO compression is being used, the following is 
> seen:
> 
> ```
> 2017-12-10 22:31:09,244|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exception in thread "main" 
> java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> com.hadoop.compression.lzo.LzoCodec
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:130)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:301)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.2017-12-10
>  22:31:09,267|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.lang.Thread.run(Thread.java:745)
> 2017-12-10 22:31:09,741|INFO|MainThread|machine.py:189 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exit Code: 1
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
>  ac71ce4b36 
> 
> 
> Diff: https://reviews.apache.org/r/64554/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing on Hbase cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64542: Kerberos support for OneFS

2017-12-13 Thread Eugene Chekanskiy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64542/#review193700
---


Ship it!




Ship It!

- Eugene Chekanskiy


On Dec. 12, 2017, 3:38 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64542/
> ---
> 
> (Updated Dec. 12, 2017, 3:38 p.m.)
> 
> 
> Review request for Ambari, Eugene Chekanskiy, Nate Cole, Robert Levas, and 
> Sandor Molnar.
> 
> 
> Bugs: AMBARI-22634
> https://issues.apache.org/jira/browse/AMBARI-22634
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When kerberos is enabled, the OneFS service check should do a kinit before 
> executing hdfs commands. I also removed some unused code from the 
> servicecheck script.
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/main/admin/kerberos/step1_controller.js 
> 9c864a8f48889515d5b72c0feaf340b7b9e1fe31 
>   ambari-web/app/messages.js b3051962f56951fc6bae169b63c5f029c1ef301d 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/kerberos.json
>  PRE-CREATION 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/package/scripts/params_linux.py
>  b823fc54b84e1149164e17504046a7a32070df1c 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/package/scripts/service_check.py
>  3d798a3342ced3d99a4a2c955a6aca27108c 
> 
> 
> Diff: https://reviews.apache.org/r/64542/diff/1/
> 
> 
> Testing
> ---
> 
> - enabled kerberos, ran service checks on onefs
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 64542: Kerberos support for OneFS

2017-12-13 Thread Robert Levas



> On Dec. 12, 2017, 11:03 a.m., Nate Cole wrote:
> > ambari-web/app/controllers/main/admin/kerberos/step1_controller.js
> > Lines 45-47 (patched)
> > 
> >
> > Is this, or can this be made to show conditionally if isilon is chosen?

+1.. We should not show this unless Isilon is involved, else it will cause some 
concusion.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64542/#review193551
---


On Dec. 12, 2017, 10:38 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64542/
> ---
> 
> (Updated Dec. 12, 2017, 10:38 a.m.)
> 
> 
> Review request for Ambari, Eugene Chekanskiy, Nate Cole, Robert Levas, and 
> Sandor Molnar.
> 
> 
> Bugs: AMBARI-22634
> https://issues.apache.org/jira/browse/AMBARI-22634
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When kerberos is enabled, the OneFS service check should do a kinit before 
> executing hdfs commands. I also removed some unused code from the 
> servicecheck script.
> 
> 
> Diffs
> -
> 
>   ambari-web/app/controllers/main/admin/kerberos/step1_controller.js 
> 9c864a8f48889515d5b72c0feaf340b7b9e1fe31 
>   ambari-web/app/messages.js b3051962f56951fc6bae169b63c5f029c1ef301d 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/kerberos.json
>  PRE-CREATION 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/package/scripts/params_linux.py
>  b823fc54b84e1149164e17504046a7a32070df1c 
>   
> contrib/management-packs/isilon-onefs-mpack/src/main/resources/addon-services/ONEFS/1.0.0/package/scripts/service_check.py
>  3d798a3342ced3d99a4a2c955a6aca27108c 
> 
> 
> Diff: https://reviews.apache.org/r/64542/diff/1/
> 
> 
> Testing
> ---
> 
> - enabled kerberos, ran service checks on onefs
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 64568: AMBARI-22638. yarn.timeline-service property enabled overriding preset secure blueprint config for the same

2017-12-13 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64568/#review193698
---


Ship it!




Ship It!

- Robert Levas


On Dec. 13, 2017, 1:54 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64568/
> ---
> 
> (Updated Dec. 13, 2017, 1:54 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Robert Levas, 
> and Robert Nettleton.
> 
> 
> Bugs: AMBARI-22638
> https://issues.apache.org/jira/browse/AMBARI-22638
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Changed `yarn.timeline-service.enabled` in `kerberos.json` to `false`, to 
> match stack default for HDP 3.0.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/kerberos.json 
> a3cfaea95918e629f02d2fa9d6eb72a5c77e5987 
> 
> 
> Diff: https://reviews.apache.org/r/64568/diff/1/
> 
> 
> Testing
> ---
> 
> Tested secure cluster deployment with HDP 3.0, verified value of 
> `yarn.timeline-service.enabled` property: `false` if unset or set to `false` 
> in blueprint, `true` if set to `true` in blueprint.
> 
> Unit tests:
> 
> ```
> $ mvn -am -pl ambari-server clean test -Dtest='kerberos/*'
> ...
> Tests run: 205, Failures: 0, Errors: 0, Skipped: 1
> ...
> Total run:1210
> Total errors:0
> Total failures:0
> ...
> BUILD SUCCESS
> ```
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Review Request 64579: Node Managers fail to start after Spark2 is patched due to CNF YarnShuffleService

2017-12-13 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64579/
---

Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.


Bugs: AMBARI-22644
https://issues.apache.org/jira/browse/AMBARI-22644


Repository: ambari


Description
---

*STR*
# Deploy HDP-2.6.4.0 cluster with Ambari-2.6.1.0-114
# Apply HBase patch Upgrade on the cluster (this step is optional)
# Then apply Spark2 patch Upgrade on the cluster
# Restart Node Managers

*Result*
NM restart fails with below error:
```
2017-12-10 07:17:02,559 INFO  impl.MetricsSystemImpl 
(MetricsSystemImpl.java:shutdown(606)) - NodeManager metrics system shutdown 
complete.
2017-12-10 07:17:02,559 FATAL nodemanager.NodeManager 
(NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManager
Caused by: java.lang.ClassNotFoundException: 
org.apache.spark.network.yarn.YarnShuffleService
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at 
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:197)
at 
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:165)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:169)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:131)
at 
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 8 more
2017-12-10 07:17:02,562 INFO  nodemanager.NodeManager 
(LogAdapter.java:info(45)) - SHUTDOWN_MSG:
```

The spark properties are correctly being written out as per AMBARI-22525.

Initially, we had defined Spark properties for ATS like this:
```xml
yarn.nodemanager.aux-services.spark_shuffle.classpath
{{stack_root}}/${hdp.version}/spark/aux/*
```

When YARN upgrades without Spark, we run into AMBARI-22525. Seems like the 
shuffle classes are installed as part of RPM dependencies, but not the 
SparkATSPlugin.

So:
- If we use YARN's version for the Spark classes, then ATS can't find 
SparkATSPlugin since that is not part of YARN.
- If we use Spark's version for the classes, then Spark can never upgrade 
without YARN since NodeManager can't find the new Spark classes. 

However, it seems like shuffle and ATS use different properties. We changed all 
3 properties in AMBARI-22525:

```
yarn.nodemanager.aux-services.spark2_shuffle.classpath
yarn.nodemanager.aux-services.spark_shuffle.classpath
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath
```

It seems like what need to do is change the spark shuffle stuff back to 
hdp.version, but leave ATS using the new version since we're guaranteed to have 
Spark installed on the ATS machine.


Diffs
-

  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
 ac71ce4b36 


Diff: https://reviews.apache.org/r/64579/diff/1/


Testing
---

Manual testing on a patched cluster with YARN/Spark


Thanks,

Jonathan Hurley

Re: Review Request 64554: HBase Cannot Find LZO Classes After Being Patched

2017-12-13 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64554/
---

(Updated Dec. 13, 2017, 10:46 a.m.)


Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.


Bugs: AMBARI-22640
https://issues.apache.org/jira/browse/AMBARI-22640


Repository: ambari


Description
---

After patching HBase where LZO compression is being used, the following is seen:

```
2017-12-10 22:31:09,244|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exception in thread "main" 
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
com.hadoop.compression.lzo.LzoCodec
2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:130)
2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:301)
2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.2017-12-10
 22:31:09,267|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
java.lang.Thread.run(Thread.java:745)
2017-12-10 22:31:09,741|INFO|MainThread|machine.py:189 - 
run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exit Code: 1
```


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
 ac71ce4b36 


Diff: https://reviews.apache.org/r/64554/diff/2/

Changes: https://reviews.apache.org/r/64554/diff/1-2/


Testing
---

Manual testing on Hbase cluster.


Thanks,

Jonathan Hurley

Re: Review Request 64554: HBase Cannot Find LZO Classes After Being Patched

2017-12-13 Thread Jonathan Hurley



> On Dec. 13, 2017, 10:04 a.m., Nate Cole wrote:
> > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
> > Lines 29-31 (original), 41-47 (patched)
> > 
> >
> > Only in WINSRV?

HAH! I just found this while testing it this morning. Fixing it now...


- Jonathan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64554/#review193680
---


On Dec. 13, 2017, 9:02 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64554/
> ---
> 
> (Updated Dec. 13, 2017, 9:02 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22640
> https://issues.apache.org/jira/browse/AMBARI-22640
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After patching HBase where LZO compression is being used, the following is 
> seen:
> 
> ```
> 2017-12-10 22:31:09,244|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exception in thread "main" 
> java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> com.hadoop.compression.lzo.LzoCodec
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:130)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:301)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.2017-12-10
>  22:31:09,267|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.lang.Thread.run(Thread.java:745)
> 2017-12-10 22:31:09,741|INFO|MainThread|machine.py:189 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exit Code: 1
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
>  ac71ce4b36 
> 
> 
> Diff: https://reviews.apache.org/r/64554/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing on Hbase cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64537: AMBARI-22635 : Ambari should create a dummy core-site.xml for Ranger plugins when namenode is not installed.

2017-12-13 Thread Vishal Suvagia via Review Board



> On Dec. 12, 2017, 4:36 p.m., Sid Wagle wrote:
> > Instead of putting empty files on the classpath isn't it easier to check 
> > for file existence on plugin side and defaul to "simple" auth?
> 
> Sid Wagle wrote:
> Correction not empty I meant dummy file.

Thank-you Siddharth, that is the current behaviour for Ranger plugins, which 
will default to simple auth if the file is not found, however once the cluster 
will be kerberized the plugin should make kerberized calls to Ranger for which 
plugin will require core-site.xml to set security as kerberos.


- Vishal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64537/#review193559
---


On Dec. 13, 2017, 3:42 p.m., Vishal Suvagia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64537/
> ---
> 
> (Updated Dec. 13, 2017, 3:42 p.m.)
> 
> 
> Review request for Ambari, Gautam Borad, Jonathan Hurley, Jayush Luniya, 
> Madhuvanthi Radhakrishnan, Mugdha Varadkar, and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-22635
> https://issues.apache.org/jira/browse/AMBARI-22635
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> For Ranger plugins to work properly in kerberised environments where HDFS is 
> not installed. We need to create a core-site.xml for Storm and Kafka plugins 
> so that the plugins can work to fetch latest policies from with kerberised 
> calls from Ranger.
> Attached patch is for trunk branch, seperate patch for branch-2.6 is attached 
> to Apache JIRA.
> 
> 
> Diffs
> -
> 
>   
> ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
>  485c1a6 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/setup_ranger_kafka.py
>  e9719aa 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/setup_ranger_kafka.py
>  9aa09df 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/setup_ranger_knox.py
>  c486ef7 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/setup_ranger_knox.py
>  c486ef7 
>   
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/setup_ranger_storm.py
>  c04496e 
>   
> ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/package/scripts/setup_ranger_storm.py
>  c04496e 
> 
> 
> Diff: https://reviews.apache.org/r/64537/diff/2/
> 
> 
> Testing
> ---
> 
> Verified with installation on Cent-OS 6.
> 
> 
> Thanks,
> 
> Vishal Suvagia
> 
>

Re: Review Request 64537: AMBARI-22635 : Ambari should create a dummy core-site.xml for Ranger plugins when namenode is not installed.

2017-12-13 Thread Vishal Suvagia via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64537/
---

(Updated Dec. 13, 2017, 3:42 p.m.)


Review request for Ambari, Gautam Borad, Jonathan Hurley, Jayush Luniya, 
Madhuvanthi Radhakrishnan, Mugdha Varadkar, and Velmurugan Periasamy.


Changes
---

Updating patch to address review comments.
Seperate patch for branch-2.6 is attached on Apache JIRA.


Bugs: AMBARI-22635
https://issues.apache.org/jira/browse/AMBARI-22635


Repository: ambari


Description
---

For Ranger plugins to work properly in kerberised environments where HDFS is 
not installed. We need to create a core-site.xml for Storm and Kafka plugins so 
that the plugins can work to fetch latest policies from with kerberised calls 
from Ranger.
Attached patch is for trunk branch, seperate patch for branch-2.6 is attached 
to Apache JIRA.


Diffs (updated)
-

  
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
 485c1a6 
  
ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/package/scripts/setup_ranger_kafka.py
 e9719aa 
  
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/setup_ranger_kafka.py
 9aa09df 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/setup_ranger_knox.py
 c486ef7 
  
ambari-server/src/main/resources/common-services/KNOX/0.5.0.3.0/package/scripts/setup_ranger_knox.py
 c486ef7 
  
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/setup_ranger_storm.py
 c04496e 
  
ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/package/scripts/setup_ranger_storm.py
 c04496e 


Diff: https://reviews.apache.org/r/64537/diff/2/

Changes: https://reviews.apache.org/r/64537/diff/1-2/


Testing
---

Verified with installation on Cent-OS 6.


Thanks,

Vishal Suvagia

Re: Review Request 63955: Validate blueprint does not allow lzo enable without setup with license agreement.

2017-12-13 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63955/#review193685
---



Has this been pushed?  If so, please close the review.

- Nate Cole


On Nov. 20, 2017, 12:06 p.m., Myroslav Papirkovskyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63955/
> ---
> 
> (Updated Nov. 20, 2017, 12:06 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, 
> Robert Nettleton, and Sid Wagle.
> 
> 
> Bugs: AMBARI-22480
> https://issues.apache.org/jira/browse/AMBARI-22480
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Blueprint validation should fail if LZO is enabled and GPL license is not 
> accepted on server setup.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintResourceProvider.java
>  1c127c080cd33314e47bcfff63e0c1e6d58937bc 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/Blueprint.java 
> c9dc7281a092ff9f22551c9f8fc64ebca15fda36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintImpl.java
>  1d917e0940b1ba5988a740005017c9d228e46f96 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidator.java
>  206d161fc694f67ae982ef668c5057b4bacec59b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java
>  1aaa5ce66d1a5385018e4f29c92441e157c2be02 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/GPLLicenseNotAcceptedException.java
>  PRE-CREATION 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintImplTest.java
>  fdbc0807788377f689e19200be9d6cdc2c992690 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintValidatorImplTest.java
>  ea6bee314192f0665d4df652cf0c401e0218026f 
> 
> 
> Diff: https://reviews.apache.org/r/63955/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Myroslav Papirkovskyy
> 
>

Re: Review Request 63957: Stack advisor should disallow lzo enable without accepting license agreement.

2017-12-13 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63957/#review193684
---



Has this been pushed?  If so, please close the review.

- Nate Cole


On Nov. 22, 2017, 1:09 p.m., Myroslav Papirkovskyy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63957/
> ---
> 
> (Updated Nov. 22, 2017, 1:09 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and 
> Sid Wagle.
> 
> 
> Bugs: AMBARI-22484
> https://issues.apache.org/jira/browse/AMBARI-22484
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> If the user adds com.hadoop.compression.lzo.LzoCodec to core-site property 
> io.compression.codec.lzo.class and the ambari.properties does not have the 
> allow GPL software install flag, throw validation error
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorRequest.java
>  64180e5039bdb74d3aaf25e140565c82a95a26c1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  5440462b86cdcfafb1e9ab239dacfddee0b31b93 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  b2a68d66d777f107fdc4f5576a691e8d25ec2c92 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackAdvisorResourceProvider.java
>  dd32c6badb18c1a8c02b7093813cd92ccd2fc74c 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> c36a41f49b224a904368c7474f0f769348e65ae6 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 
> 8ccbaba9db083068268951ae4e07ce08d32edcc4 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ValidationResourceProviderTest.java
>  cdbf2d782f86b2a30f54619c4e2a9165a020275c 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> 9e0070458987ee490f3918fead5ef11d201c8612 
> 
> 
> Diff: https://reviews.apache.org/r/63957/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Myroslav Papirkovskyy
> 
>

Re: Review Request 64553: Log Feeder refactor: integrate with spring boot

2017-12-13 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64553/#review193683
---


Ship it!




Ship It!

- Krisztian Kasa


On Dec. 13, 2017, 3:24 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64553/
> ---
> 
> (Updated Dec. 13, 2017, 3:24 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa and Sid Wagle.
> 
> 
> Bugs: AMBARI-22639
> https://issues.apache.org/jira/browse/AMBARI-22639
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> First patch for refactor logfeeder - Spring Boot support
> 
> Add spring boot support, the goal is to get rid of LogFeederPropertiesUtil 
> and let Spring to handle configs & beans.
> - remove LogFeederPropertiesUtil
> - move logfeeder properties fields into mutliple POJOs
> - update some tests & remove one test (needs to rewrite it)
> - convert singleton classes to spring beans
> - let Spring to handle application pid file
> - convert some internal threads to classes (like StatsLogger)
> 
> (it does not contain any real changes regarding to input-filter-output blocks)
> 
> Next patch will be focus on the usage of start scritps (in ambari-logsearch 
> and ambari-server stack code)
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
>  a84a97b 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java
>  6ed36fd 
>   ambari-logsearch/ambari-logsearch-logfeeder/.gitignore PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 01710bf 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  5114743 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeederCommandLine.java
>  d996f98 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
>  243b344 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigItem.java
>  5c20a8e 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
>  ec29f69 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
>  a76 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/ApplicationConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/InputSimulateConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogEntryCacheConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/MetricsCollectorConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/Filter.java
>  8e8834b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java
>  fc7a565 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterKeyValue.java
>  8e5aee8 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
>  b021c37 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
>  972011d 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
>  8f8c4fd 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
>  f1b422f 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java
>  df6c941 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/loglevelfilter/FilterLogData.java
>  6173f53 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/loglevelfilter/LogLevelFilterHandler.java
>  e44873b 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeede

Re: Review Request 64553: Log Feeder refactor: integrate with spring boot

2017-12-13 Thread Oliver Szabo



> On Dec. 13, 2017, 9:22 a.m., Krisztian Kasa wrote:
> > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigItem.java
> > Line 62 (original), 64 (patched)
> > 
> >
> > Create an abstract init method and call it from here. The comment "This 
> > method needs to be overwritten by deriving classes." would be unnecessary.
> > Like: 
> > 
> > protected abstract onInitializing(LogFeederProps logFeederProps);
> > 
> > public void init(LogFeederProps logFeederProps) throws Exception {
> > this.logFeederProps = logFeederProps;
> > onInitializing(logFeederProps);
> > }

you are right but as i wrote it does not contain any real changes on the blocks 
part (focus is on make it work with spring boot), so these will be refactored 
in different patch


> On Dec. 13, 2017, 9:22 a.m., Krisztian Kasa wrote:
> > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
> > Lines 213 (patched)
> > 
> >
> > Are the same properties are stored here as the fields of this class?

no its all properties, so you can pass all of them if you want through this 
class


> On Dec. 13, 2017, 9:22 a.m., Krisztian Kasa wrote:
> > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java
> > Lines 160 (patched)
> > 
> >
> > Isn't it isEmpty?

yes, thanks


> On Dec. 13, 2017, 9:22 a.m., Krisztian Kasa wrote:
> > ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/StatsLogger.java
> > Lines 64 (patched)
> > 
> >
> > Wouldn't be better to halt the entire application in case of an Error? 
> > So Errors would be rethrown but Exceptions not.

regarding the 2 statLogger comment: probably it will be refactored much more 
(or i will get rid of this), but now i just moved the class (but review board 
show as a newly added one)


- Oliver


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64553/#review193647
---


On Dec. 13, 2017, 3:24 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64553/
> ---
> 
> (Updated Dec. 13, 2017, 3:24 p.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa and Sid Wagle.
> 
> 
> Bugs: AMBARI-22639
> https://issues.apache.org/jira/browse/AMBARI-22639
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> First patch for refactor logfeeder - Spring Boot support
> 
> Add spring boot support, the goal is to get rid of LogFeederPropertiesUtil 
> and let Spring to handle configs & beans.
> - remove LogFeederPropertiesUtil
> - move logfeeder properties fields into mutliple POJOs
> - update some tests & remove one test (needs to rewrite it)
> - convert singleton classes to spring beans
> - let Spring to handle application pid file
> - convert some internal threads to classes (like StatsLogger)
> 
> (it does not contain any real changes regarding to input-filter-output blocks)
> 
> Next patch will be focus on the usage of start scritps (in ambari-logsearch 
> and ambari-server stack code)
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
>  a84a97b 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java
>  6ed36fd 
>   ambari-logsearch/ambari-logsearch-logfeeder/.gitignore PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 01710bf 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  5114743 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeederCommandLine.java
>  d996f98 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
>  243b344 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigItem.java
>  5c20a8e 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
>  ec29f69 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
>  a76 
>   
> ambari-logsearch/

Re: Review Request 64553: Log Feeder refactor: integrate with spring boot

2017-12-13 Thread Oliver Szabo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64553/
---

(Updated Dec. 13, 2017, 3:24 p.m.)


Review request for Ambari, Krisztian Kasa and Sid Wagle.


Changes
---

fix issues


Bugs: AMBARI-22639
https://issues.apache.org/jira/browse/AMBARI-22639


Repository: ambari


Description
---

First patch for refactor logfeeder - Spring Boot support

Add spring boot support, the goal is to get rid of LogFeederPropertiesUtil and 
let Spring to handle configs & beans.
- remove LogFeederPropertiesUtil
- move logfeeder properties fields into mutliple POJOs
- update some tests & remove one test (needs to rewrite it)
- convert singleton classes to spring beans
- let Spring to handle application pid file
- convert some internal threads to classes (like StatsLogger)

(it does not contain any real changes regarding to input-filter-output blocks)

Next patch will be focus on the usage of start scritps (in ambari-logsearch and 
ambari-server stack code)


Diffs (updated)
-

  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
 a84a97b 
  
ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java
 6ed36fd 
  ambari-logsearch/ambari-logsearch-logfeeder/.gitignore PRE-CREATION 
  ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 01710bf 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
 5114743 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeederCommandLine.java
 d996f98 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java
 243b344 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigItem.java
 5c20a8e 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
 ec29f69 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
 a76 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/ApplicationConfig.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/InputSimulateConfig.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogEntryCacheConfig.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/MetricsCollectorConfig.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/Filter.java
 8e8834b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java
 fc7a565 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterKeyValue.java
 8e5aee8 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java
 b021c37 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java
 972011d 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java
 8f8c4fd 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManager.java
 f1b422f 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSimulate.java
 df6c941 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/loglevelfilter/FilterLogData.java
 6173f53 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/loglevelfilter/LogLevelFilterHandler.java
 e44873b 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/LogFeederAMSClient.java
 c832358 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/MetricsManager.java
 6e8ac04 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/StatsLogger.java
 PRE-CREATION 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputFile.java
 e1a0bb9 
  
ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputHDFSFile.java

Re: Review Request 64554: HBase Cannot Find LZO Classes After Being Patched

2017-12-13 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64554/#review193680
---




ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
Lines 29-31 (original), 41-47 (patched)


Only in WINSRV?


- Nate Cole


On Dec. 13, 2017, 9:02 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64554/
> ---
> 
> (Updated Dec. 13, 2017, 9:02 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, and Nate Cole.
> 
> 
> Bugs: AMBARI-22640
> https://issues.apache.org/jira/browse/AMBARI-22640
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After patching HBase where LZO compression is being used, the following is 
> seen:
> 
> ```
> 2017-12-10 22:31:09,244|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exception in thread "main" 
> java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> com.hadoop.compression.lzo.LzoCodec
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:130)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:301)
> 2017-12-10 22:31:09,245|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.2017-12-10
>  22:31:09,267|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2017-12-10 22:31:09,268|INFO|MainThread|machine.py:164 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|at 
> java.lang.Thread.run(Thread.java:745)
> 2017-12-10 22:31:09,741|INFO|MainThread|machine.py:189 - 
> run()||GUID=37b565a7-e164-4641-b335-19884c614ffd|Exit Code: 1
> ```
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase.py
>  ac71ce4b36 
> 
> 
> Diff: https://reviews.apache.org/r/64554/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing on Hbase cluster.
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 64571: AMBARI-22404: Set java.home. in ambari-server setup script for server OS family

2017-12-13 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64571/#review193682
---




ambari-server/src/main/python/ambari_server/serverSetup.py
Lines 870-876 (patched)


This appears to always force someone to pick an OS and changes the 
expectation of how the server gets installed.

We shouldn't be changing how things are working today.  You should do 
either:
- After picking JAVA_HOME, then we should ask if they want to customize the 
OS type (as a choice).
- Use another command line option to add a JAVA_HOME for an OS type.  No 
need to ask the type, just use whatever value they put in the option.  This 
way, you only have to check for the option to know to write 
"java.home.redhat7".  No option means we maintain current behavior.

I would prefer the 2nd option as it's likely easier to document.  If we 
start asking "hard choices" the everyday user will get confused.


- Nate Cole


On Dec. 13, 2017, 4:28 a.m., Pravin Dsilva wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64571/
> ---
> 
> (Updated Dec. 13, 2017, 4:28 a.m.)
> 
> 
> Review request for Ambari and Nate Cole.
> 
> 
> Bugs: AMBARI-22404
> https://issues.apache.org/jira/browse/AMBARI-22404
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Changes to ambari-server setup script for accepting user values for Java Home 
> for OS type.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/python/ambari_server/serverSetup.py 13d5348 
>   ambari-server/src/test/python/TestAmbariServer.py 8ce2457 
> 
> 
> Diff: https://reviews.apache.org/r/64571/diff/1/
> 
> 
> Testing
> ---
> 
> All tests passed
> 
> 
> Thanks,
> 
> Pravin Dsilva
> 
>

Re: Review Request 63786: zookeeper service issue

2017-12-13 Thread zhangxiaolu zhangxiaolu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63786/
---

(Updated 十二月 13, 2017, 11:36 a.m.)


Review request for Ambari, Andrii Babiichuk, Alejandro Fernandez, Denys Buzhor, 
Dmitro Lisnichenko, Dmytro Sen, Jaimin Jetly, Nate Cole, Richard Zang, Srimanth 
Gunturi, Sid Wagle, Vitalyi Brodetskyi, and Yusaku Sako.


Bugs: AMBARI-22436
https://issues.apache.org/jira/browse/AMBARI-22436


Repository: ambari


Description (updated)
---

(1)amabri-qa has not permission to write zookeeper'.log in the dir of 
/var/log/zookeeper for permission is 0755.
so should set permission is 0777 when executing service check.
(2)check can pass but there has some warns:java.io.FileNotFoundException: 
/var/log/zookeeper/zookeeper.log (Permission denied)


Diffs
-

  
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/service_check.py
 b7c7758 


Diff: https://reviews.apache.org/r/63786/diff/1/


Testing
---

done


File Attachments (updated)


AMBARI-22436_1.patch
  
https://reviews.apache.org/media/uploaded/files/2017/12/13/4c703fee-d7de-4363-92c0-4e6ec4094bed__AMBARI-22436_1.patch


Thanks,

zhangxiaolu zhangxiaolu

Re: Review Request 64204: zkfc show no alerts when zkfc is down

2017-12-13 Thread zhangxiaolu zhangxiaolu



> On 十二月 4, 2017, 4:24 p.m., Nate Cole wrote:
> > ambari-web/app/mappers/alert_definition_summary_mapper.js
> > Lines 94-106 (patched)
> > 
> >
> > Strange that this isn't getting covered in the above block.  I'm not a 
> > UI engineer, but seems like special casing ZKFC isn't a good idea and that 
> > any SLAVE-type should be presenting it's alert data?

Yes,We stopped all components and there shows no alerts, Actually, when we 
opened the tip, there were two critical alerts. Maybe this method is not the 
best, we can change it.


- zhangxiaolu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64204/#review192706
---


On 十一月 30, 2017, 11:40 a.m., zhangxiaolu zhangxiaolu wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64204/
> ---
> 
> (Updated 十一月 30, 2017, 11:40 a.m.)
> 
> 
> Review request for Ambari, Andrii Babiichuk, Alejandro Fernandez, Denys 
> Buzhor, Dmitro Lisnichenko, Dmytro Sen, Jaimin Jetly, Nate Cole, Richard 
> Zang, Srimanth Gunturi, Sid Wagle, Vitalyi Brodetskyi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-22557
> https://issues.apache.org/jira/browse/AMBARI-22557
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> (1)zkfc show no alerts when zkfc is down
> (2)we should deal with zkfc in front.
> 
> 
> Diffs
> -
> 
>   ambari-web/app/mappers/alert_definition_summary_mapper.js 29f286e 
> 
> 
> Diff: https://reviews.apache.org/r/64204/diff/1/
> 
> 
> Testing
> ---
> 
> done it
> 
> 
> File Attachments
> 
> 
> AMBARI-22557.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/11/30/77e10ef5-6f44-4e18-9c8f-20d9fe62d8a9__AMBARI-22557.patch
> 
> 
> Thanks,
> 
> zhangxiaolu zhangxiaolu
> 
>

Review Request 64571: AMBARI-22404: Set java.home. in ambari-server setup script for server OS family

2017-12-13 Thread Pravin Dsilva


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64571/
---

Review request for Ambari and Nate Cole.


Bugs: AMBARI-22404
https://issues.apache.org/jira/browse/AMBARI-22404


Repository: ambari


Description
---

Changes to ambari-server setup script for accepting user values for Java Home 
for OS type.


Diffs
-

  ambari-server/src/main/python/ambari_server/serverSetup.py 13d5348 
  ambari-server/src/test/python/TestAmbariServer.py 8ce2457 


Diff: https://reviews.apache.org/r/64571/diff/1/


Testing
---

All tests passed


Thanks,

Pravin Dsilva

Re: Review Request 64553: Log Feeder refactor: integrate with spring boot

2017-12-13 Thread Krisztian Kasa


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64553/#review193647
---




ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigItem.java
Line 62 (original), 64 (patched)


Create an abstract init method and call it from here. The comment "This 
method needs to be overwritten by deriving classes." would be unnecessary.
Like: 

protected abstract onInitializing(LogFeederProps logFeederProps);

public void init(LogFeederProps logFeederProps) throws Exception {
this.logFeederProps = logFeederProps;
onInitializing(logFeederProps);
}



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogEntryCacheConfig.java
Lines 41 (patched)


typo: repteitions -> repetitions



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
Lines 77 (patched)


typo: separtaed -> separated



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
Lines 115 (patched)


typo: wher -> where



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
Lines 213 (patched)


Are the same properties are stored here as the fields of this class?



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java
Lines 138 (patched)


isBlank checks if the string is null or empty string or whitespace only. It 
can be useful instead of isEmpty if you have to deal with whitespaces.



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java
Lines 160 (patched)


Isn't it isEmpty?



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/MetricsCollectorConfig.java
Lines 105 (patched)


!StringUtils.isBlank(hostsString) can be used. It also performs null check.



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/StatsLogger.java
Lines 59 (patched)


Wouldn't be enough catching InterruptedException only?



ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/StatsLogger.java
Lines 64 (patched)


Wouldn't be better to halt the entire application in case of an Error? So 
Errors would be rethrown but Exceptions not.


- Krisztian Kasa


On Dec. 13, 2017, 1:50 a.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64553/
> ---
> 
> (Updated Dec. 13, 2017, 1:50 a.m.)
> 
> 
> Review request for Ambari, Krisztian Kasa and Sid Wagle.
> 
> 
> Bugs: AMBARI-22639
> https://issues.apache.org/jira/browse/AMBARI-22639
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> First patch for refactor logfeeder - Spring Boot support
> 
> Add spring boot support, the goal is to get rid of LogFeederPropertiesUtil 
> and let Spring to handle configs & beans.
> - remove LogFeederPropertiesUtil
> - move logfeeder properties fields into mutliple POJOs
> - update some tests & remove one test (needs to rewrite it)
> - convert singleton classes to spring beans
> - let Spring to handle application pid file
> - convert some internal threads to classes (like StatsLogger)
> 
> (it does not contain any real changes regarding to input-filter-output blocks)
> 
> Next patch will be focus on the usage of start scritps (in ambari-logsearch 
> and ambari-server stack code)
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigFactory.java
>  a84a97b 
>   
> ambari-logsearch/ambari-logsearch-config-api/src/main/java/org/apache/ambari/logsearch/config/api/LogSearchConfigLogFeeder.java
>  6ed36fd 
>   ambari-logsearch/ambari-logsearch-logfeeder/.gitignore PRE-CREATION 
>   ambari-logsearch/ambari-logsearch-logfeeder/pom.xml 01710bf 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/LogFeeder.java
>  5114743 
>   
> ambari-logsearch/ambari-logsea

37 matches

Mail list logo