date:20170821

Re: Review Request 61501: Prevent users from authenticating if they exceed a configured number of login failures

2017-08-21 Thread Attila Magyar



> On Aug. 10, 2017, 6:26 p.m., Robert Levas wrote:
> > Would it have been possble to add the lockout logic in 
> > `org.apache.ambari.server.security.authentication.AmbariAuthenticationEventHandlerImpl#onSuccessfulAuthentication`?
> >   I am not sure if chaning the _success_ to a _failure_ is possible here, 
> > though.
> > 
> > If the code stays as is, does the consective failure count increase when 
> > the user fails to authenticate becuase the failure count was passed the 
> > threshhold? If so, do we care?
> 
> Attila Magyar wrote:
> First I wanted to put it there, but I didn't find a way to move to the 
> failure case from there.
> 
> The counter will keep increasing after it passed the threashold. This can 
> lead to a possible overflow after ~2 billion failures, which will eventually 
> reset the counter. So maybe we shouldn't increase it beyond a certain limit. 
> I can add overflow detection to UserEntity>incrementConsecutiveFailures and 
> maximize the counter to Integer.MAX_VALUE. Or do you have a better idea?
> 
> Robert Levas wrote:
> Can we check the AmbariAuthenticationException value and not increment if 
> the cause is/contains a TooManyLoginFailuresException?
> 
> Attila Magyar wrote:
> The number of consecutive failures is shown on the UI, wouldn't it be 
> confusing to display a number that doesn't reflect the real number of 
> failures?
> 
> Robert Levas wrote:
> Ideally, the login faulure does not indicate that the user is locked out. 
>  The error message should be the same as any other authentication failure.  
> We don't want to give away any information about the user account in the even 
> someone is trying to brute force their way in.  
> 
> If we are telling the user that they have failed to authenticate X number 
> of times, then constantly incremeneting the failure count is preferred - no 
> matter if the user was locked out or not. Therefore the current 
> implementation is correct.. however maybe a little note somewhere in the code 
> would be nice to have to make sure no one "fixes" the issue potentially 
> allowing a hacker to know they have successfully authenticated to a locked 
> account.

The number of consecutive login failures is only displayed on the admin page 
after you logged in, next to the reset button.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61501/#review182616
---


On Aug. 11, 2017, 7:32 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61501/
> ---
> 
> (Updated Aug. 11, 2017, 7:32 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Robert Levas, 
> and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21680
> https://issues.apache.org/jira/browse/AMBARI-21680
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Prevent users from authenticating if they exceed a configured number of login 
> failures, which is set as a configuration in the ambari.properties file - 
> authentication.max.failures.
> 
> After a users successfully authenticates, check the value of 
> org.apache.ambari.server.orm.entities.UserEntity#getConsecutiveFailures. 
> 
> If it exceeds the value set in authentication.max.failures, then fail 
> authentication. Else allow authentication to proceed.
> If failing authentication due to being "locked out", do not indicate this to 
> the user; however an Ambari server log message will be useful. 
> The normal "authentication failed" message should be returned as to not give 
> away any information about a user's authentication. If a special "locked out" 
> message is shown, then a hacker will be able to attempt a brute force attack 
> on a user's account since the returned error message will be different if 
> they eventually succeed in guessing the password.
> 
> To "unlock" the user, a user administrator (a user with the 
> AMBARI.MANAGE_USERS authorization) needs to reset the user's consecutive 
> failure count to 0.
> By default the authentication.max.failures should be 10; however 0 should 
> indicate that no lockout is desired.
> Options
> 
> 
> Diffs
> -
> 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/users/UsersShowCtrl.js
>  200872e 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/i18n.config.js 
> 43b32da 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/User.js 
> ac50653 
>   ambari-admin/src/main/resources/ui/admin-web/app/views/users/show.html 
> f965c5d 
>   ambari-server/docs/configuration/index.md 9dbe9c4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  4f787c6 
>   
> ambari-server/src/main/

Re: Review Request 61205: Users randomly getting "HDFS020 Could not write file" exceptions while running query from Hive View

2017-08-21 Thread venkat sairam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61205/
---

(Updated Aug. 21, 2017, 8:28 a.m.)


Review request for Ambari, Gaurav Nagar, Nitiraj Rathore, Pallav Kulshreshtha, 
and Rohit Choudhary.


Bugs: AMBARI-21569
https://issues.apache.org/jira/browse/AMBARI-21569


Repository: ambari


Description
---

ServiceFormattedException:100 - 
org.apache.ambari.view.utils.hdfs.HdfsApiException: HDFS020 Could not write 
file /user/user/hive/jobs/hive-job-5290-2017-03-22_02-35/logs

org.apache.ambari.view.utils.hdfs.HdfsApiException: HDFS020 Could not write 
file /user/user/hive/jobs/hive-job-5290-2017-03-22_02-35/logs
at 
org.apache.ambari.view.utils.hdfs.HdfsUtil.putStringToFile(HdfsUtil.java:51)
at 
org.apache.ambari.view.hive.resources.jobs.viewJobs.JobControllerImpl.updateOperationLogs(JobControllerImpl.java:202)
at 
org.apache.ambari.view.hive.resources.jobs.viewJobs.JobControllerImpl.update(JobControllerImpl.java:160)
at 
org.apache.ambari.view.hive.resources.jobs.viewJobs.JobResourceManager.read(JobResourceManager.java:79)
at 
org.apache.ambari.view.hive.resources.jobs.viewJobs.JobResourceManager.readController(JobResourceManager.java:95)
at 
org.apache.ambari.view.hive.resources.jobs.JobService.getOne(JobService.java:133)
at sun.reflect.GeneratedMethodAccessor191.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at 
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at 
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at 
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at 
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at 
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at 
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at 
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at 
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at 
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at 
org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:257)
at 
org.springframework.securi

Re: Review Request 61501: Prevent users from authenticating if they exceed a configured number of login failures

2017-08-21 Thread Robert Levas



> On Aug. 10, 2017, 2:26 p.m., Robert Levas wrote:
> > Would it have been possble to add the lockout logic in 
> > `org.apache.ambari.server.security.authentication.AmbariAuthenticationEventHandlerImpl#onSuccessfulAuthentication`?
> >   I am not sure if chaning the _success_ to a _failure_ is possible here, 
> > though.
> > 
> > If the code stays as is, does the consective failure count increase when 
> > the user fails to authenticate becuase the failure count was passed the 
> > threshhold? If so, do we care?
> 
> Attila Magyar wrote:
> First I wanted to put it there, but I didn't find a way to move to the 
> failure case from there.
> 
> The counter will keep increasing after it passed the threashold. This can 
> lead to a possible overflow after ~2 billion failures, which will eventually 
> reset the counter. So maybe we shouldn't increase it beyond a certain limit. 
> I can add overflow detection to UserEntity>incrementConsecutiveFailures and 
> maximize the counter to Integer.MAX_VALUE. Or do you have a better idea?
> 
> Robert Levas wrote:
> Can we check the AmbariAuthenticationException value and not increment if 
> the cause is/contains a TooManyLoginFailuresException?
> 
> Attila Magyar wrote:
> The number of consecutive failures is shown on the UI, wouldn't it be 
> confusing to display a number that doesn't reflect the real number of 
> failures?
> 
> Robert Levas wrote:
> Ideally, the login faulure does not indicate that the user is locked out. 
>  The error message should be the same as any other authentication failure.  
> We don't want to give away any information about the user account in the even 
> someone is trying to brute force their way in.  
> 
> If we are telling the user that they have failed to authenticate X number 
> of times, then constantly incremeneting the failure count is preferred - no 
> matter if the user was locked out or not. Therefore the current 
> implementation is correct.. however maybe a little note somewhere in the code 
> would be nice to have to make sure no one "fixes" the issue potentially 
> allowing a hacker to know they have successfully authenticated to a locked 
> account.
> 
> Attila Magyar wrote:
> The number of consecutive login failures is only displayed on the admin 
> page after you logged in, next to the reset button.

Great... thanks for the clarification.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61501/#review182616
---


On Aug. 11, 2017, 3:32 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61501/
> ---
> 
> (Updated Aug. 11, 2017, 3:32 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Robert Levas, 
> and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21680
> https://issues.apache.org/jira/browse/AMBARI-21680
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Prevent users from authenticating if they exceed a configured number of login 
> failures, which is set as a configuration in the ambari.properties file - 
> authentication.max.failures.
> 
> After a users successfully authenticates, check the value of 
> org.apache.ambari.server.orm.entities.UserEntity#getConsecutiveFailures. 
> 
> If it exceeds the value set in authentication.max.failures, then fail 
> authentication. Else allow authentication to proceed.
> If failing authentication due to being "locked out", do not indicate this to 
> the user; however an Ambari server log message will be useful. 
> The normal "authentication failed" message should be returned as to not give 
> away any information about a user's authentication. If a special "locked out" 
> message is shown, then a hacker will be able to attempt a brute force attack 
> on a user's account since the returned error message will be different if 
> they eventually succeed in guessing the password.
> 
> To "unlock" the user, a user administrator (a user with the 
> AMBARI.MANAGE_USERS authorization) needs to reset the user's consecutive 
> failure count to 0.
> By default the authentication.max.failures should be 10; however 0 should 
> indicate that no lockout is desired.
> Options
> 
> 
> Diffs
> -
> 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/users/UsersShowCtrl.js
>  200872e 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/i18n.config.js 
> 43b32da 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/User.js 
> ac50653 
>   ambari-admin/src/main/resources/ui/admin-web/app/views/users/show.html 
> f965c5d 
>   ambari-server/docs/configuration/index.md 9dbe9c4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/co

Re: Review Request 61501: Prevent users from authenticating if they exceed a configured number of login failures

2017-08-21 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61501/#review183313
---


Ship it!




Ship It!

- Robert Levas


On Aug. 11, 2017, 3:32 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61501/
> ---
> 
> (Updated Aug. 11, 2017, 3:32 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Robert Levas, 
> and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21680
> https://issues.apache.org/jira/browse/AMBARI-21680
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Prevent users from authenticating if they exceed a configured number of login 
> failures, which is set as a configuration in the ambari.properties file - 
> authentication.max.failures.
> 
> After a users successfully authenticates, check the value of 
> org.apache.ambari.server.orm.entities.UserEntity#getConsecutiveFailures. 
> 
> If it exceeds the value set in authentication.max.failures, then fail 
> authentication. Else allow authentication to proceed.
> If failing authentication due to being "locked out", do not indicate this to 
> the user; however an Ambari server log message will be useful. 
> The normal "authentication failed" message should be returned as to not give 
> away any information about a user's authentication. If a special "locked out" 
> message is shown, then a hacker will be able to attempt a brute force attack 
> on a user's account since the returned error message will be different if 
> they eventually succeed in guessing the password.
> 
> To "unlock" the user, a user administrator (a user with the 
> AMBARI.MANAGE_USERS authorization) needs to reset the user's consecutive 
> failure count to 0.
> By default the authentication.max.failures should be 10; however 0 should 
> indicate that no lockout is desired.
> Options
> 
> 
> Diffs
> -
> 
>   
> ambari-admin/src/main/resources/ui/admin-web/app/scripts/controllers/users/UsersShowCtrl.js
>  200872e 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/i18n.config.js 
> 43b32da 
>   ambari-admin/src/main/resources/ui/admin-web/app/scripts/services/User.js 
> ac50653 
>   ambari-admin/src/main/resources/ui/admin-web/app/views/users/show.html 
> f965c5d 
>   ambari-server/docs/configuration/index.md 9dbe9c4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  4f787c6 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  b52e2b1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/UserRequest.java
>  d0836a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UserResourceProvider.java
>  a2d9917 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/AmbariAuthenticationEventHandlerImpl.java
>  3a5a66b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/TooManyLoginFailuresException.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLocalUserProvider.java
>  2c8bf12 
>   
> ambari-server/src/test/java/org/apache/ambari/server/security/authorization/AmbariLocalUserProviderTest.java
>  133fc9f 
> 
> 
> Diff: https://reviews.apache.org/r/61501/diff/3/
> 
> 
> Testing
> ---
> 
> 1
> - tried to log in 10 times with an incorrect password
> - tried to log in with the correct password and received authentication 
> failure
> - reseted the counter
> - logged in successfully with the correct password
> 
> 2
> - tried to log in 3 times with incorrect password
> - checked that UI showed Login failures=3
> - logged in successfully with the correct password
> - checked the UI showed Login failures=0
> 
> 3
> - added authentication.max.failures=3 to ambari.properties
> - repeated the first test
> 
> 4
> - added authentication.max.failures=0 to ambari.properties
> - tried to log in 11 times with an incorrect password
> - successfully logged in with the correct password
> 
> Results :
> Tests run: 4841, Failures: 0, Errors: 0, Skipped: 33
> --
> Total run:1145
> Total errors:0
> Total failures:0
> OK
> Ran 470 tests in 108.068s
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 61746: Prevent New Clusters from Being Provisioned With PATCH/MAINT Repos

2017-08-21 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61746/#review183315
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On Aug. 19, 2017, 4:23 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61746/
> ---
> 
> (Updated Aug. 19, 2017, 4:23 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Nate Cole, 
> and Robert Levas.
> 
> 
> Bugs: AMBARI-21758
> https://issues.apache.org/jira/browse/AMBARI-21758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> New clusters should only be able to be provisioned using a {{STANDARD}} 
> repository. 
> 
> STR:
> - Register a 2.5.0.0-1234 {{STANDARD}} repo
> - Register a 2.5.4.0- {{MAINT} repo
> - Use a blueprint go create a repo for 2.5.4.0-
> 
> The cluster is created and is in a weird state. We should reject these types 
> of repos...
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  60f5ccea67 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ClusterRequest.java
>  aea2072c65 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceResourceProvider.java
>  2b4bf3bba9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
>  27b1654fd3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  d77d13c00b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  fc4fa868b5 
> 
> 
> Diff: https://reviews.apache.org/r/61746/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 61743: Use latest-vdf for default when version is unspecified

2017-08-21 Thread Sebastian Toader


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61743/#review183316
---


Ship it!




Ship It!

- Sebastian Toader


On Aug. 18, 2017, 6:03 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61743/
> ---
> 
> (Updated Aug. 18, 2017, 6:03 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Sandor Magyari, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21756
> https://issues.apache.org/jira/browse/AMBARI-21756
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Use a new construction in hdp_urlinfo.json ("latest-vdf") that indicates what 
> the latest release is for a stack.  This will be used in Blueprints when the 
> version is NOT known NOR added to the DB before-hand.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  2cecfb650a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
>  ea592e50af 
>   ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
> ddc7c60ece 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/repository/VersionDefinitionXml.java
>  45d8e8e196 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/stack/LatestRepoCallable.java
>  3c7c0016ef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  d77d13c00b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/api/services/AmbariMetaInfoTest.java
>  63cb21ffa8 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java
>  0809d63968 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProviderTest.java
>  cb25aecdeb 
>   
> ambari-server/src/test/java/org/apache/ambari/server/stack/StackManagerTest.java
>  c9d3d2b7e2 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  fc4fa868b5 
>   ambari-server/src/test/resources/stacks/HDP/2.2.0/repos/hdp.json 708d757fe5 
>   ambari-server/src/test/resources/stacks/HDP/2.2.0/repos/repoinfo.xml 
> 031f5fd28a 
>   ambari-server/src/test/resources/stacks/HDP/2.2.0/repos/version-2.2.0.5.xml 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/HDP/2.2.1/metainfo.xml PRE-CREATION 
>   ambari-server/src/test/resources/stacks/HDP/2.2.1/repos/hdp.json 
> PRE-CREATION 
>   ambari-server/src/test/resources/stacks/HDP/2.2.1/repos/repoinfo.xml 
> PRE-CREATION 
>   
> ambari-server/src/test/resources/stacks/HDP/2.2.1/services/RANGER/alerts.json 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61743/diff/2/
> 
> 
> Testing
> ---
> 
> Manual.  Automated:
> 
> [INFO] Results:
> [INFO]
> [WARNING] Tests run: 4818, Failures: 0, Errors: 0, Skipped: 34
> [INFO]
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 23:56.608s
> [INFO] Finished at: Fri Aug 18 13:34:04 EDT 2017
> [INFO] Final Memory: 30M/1991M
> [INFO] 
> 
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 61707: Pre-configure services when Kerberos is enabled to reduce number of core service restarts when services are added

2017-08-21 Thread Sebastian Toader


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61707/#review183323
---


Fix it, then Ship it!




```HDP/2.6/kerberos_preconfigure.json``` has to be added to ```HDP/3.0/``` as 
well.


ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
Lines 1307-1310 (original), 1311-1324 (patched)


This can be simplified by always executing 
```readKerberosDescriptorFromFile(stackInfo.getKerberosDescriptorFileLocation())```
 and than update the descriptor with preconfigured kerberos descriptor if 
defined.



ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
Lines 1322 (patched)


Small typo: "cata" -> "data"


- Sebastian Toader


On Aug. 17, 2017, 2:51 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61707/
> ---
> 
> (Updated Aug. 17, 2017, 2:51 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene 
> Chekanskiy, Jonathan Hurley, Laszlo Puskas, Nate Cole, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21602
> https://issues.apache.org/jira/browse/AMBARI-21602
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Pre-configure (certain) services when Kerberos is enabled to reduce number of 
> core service restarts when services are added.  
> 
> While processing the Kerberos descriptor, include services marked to be 
> _pre-configured_.  When a tagged service is encountered, process it weather 
> it is installed or not. However if it is not installed, only apply 
> configuration changes for existing configuration types.  This will set at 
> least the core-site changes related to proxyuser and auth-to-local rules 
> properties. By doing this, if a tagged service is later installed, the 
> settings will already be in place in the existing service configs and thus 
> the existing services will not need to be restarted. 
> 
> Caveats:
> - Default values for the uninstalled, tagged, services will be assumed 
> - The stack advisor will be used to suggest locations of components - used to 
> build the clusterHostInfo structure that may be used to derive property 
> values. 
> 
> Note: This processing is to occur when Kerberos is enabled.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  eb97ee376d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorRequest.java
>  7ba1b1887d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java
>  3329e76226 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
>  3819863763 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  6c6c43911b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  91a84ea643 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterKerberosDescriptorResourceProvider.java
>  59bd96a8f5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackVersionResourceProvider.java
>  64ead405eb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/utilities/RemovableIdentities.java
>  d4bb501231 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/utilities/UsedIdentities.java
>  46f5642eb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  dd2b2237e5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosServerAction.java
>  2e331bb77f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PreconfigureServiceType.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareDisableKerberosServerAction.java
>  60523cdd75 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareEnableKerberosServerAction.java
>  ca15695564 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareKerberosIdentitiesServerAction.java
>  f239cffd93 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
>  78aaa77a48 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java
>  23fd0a95f5 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> ff1d37808d 
>   ambari-server/

Re: Review Request 61746: Prevent New Clusters from Being Provisioned With PATCH/MAINT Repos

2017-08-21 Thread Dmytro Grinenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61746/#review183324
---


Ship it!




Ship It!

- Dmytro Grinenko


On Aug. 19, 2017, 1:23 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61746/
> ---
> 
> (Updated Aug. 19, 2017, 1:23 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Nate Cole, 
> and Robert Levas.
> 
> 
> Bugs: AMBARI-21758
> https://issues.apache.org/jira/browse/AMBARI-21758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> New clusters should only be able to be provisioned using a {{STANDARD}} 
> repository. 
> 
> STR:
> - Register a 2.5.0.0-1234 {{STANDARD}} repo
> - Register a 2.5.4.0- {{MAINT} repo
> - Use a blueprint go create a repo for 2.5.4.0-
> 
> The cluster is created and is in a weird state. We should reject these types 
> of repos...
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  60f5ccea67 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ClusterRequest.java
>  aea2072c65 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceResourceProvider.java
>  2b4bf3bba9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
>  27b1654fd3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  d77d13c00b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  fc4fa868b5 
> 
> 
> Diff: https://reviews.apache.org/r/61746/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 61746: Prevent New Clusters from Being Provisioned With PATCH/MAINT Repos

2017-08-21 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61746/#review183325
---


Ship it!




Ship It!

- Nate Cole


On Aug. 18, 2017, 9:23 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61746/
> ---
> 
> (Updated Aug. 18, 2017, 9:23 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Nate Cole, 
> and Robert Levas.
> 
> 
> Bugs: AMBARI-21758
> https://issues.apache.org/jira/browse/AMBARI-21758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> New clusters should only be able to be provisioned using a {{STANDARD}} 
> repository. 
> 
> STR:
> - Register a 2.5.0.0-1234 {{STANDARD}} repo
> - Register a 2.5.4.0- {{MAINT} repo
> - Use a blueprint go create a repo for 2.5.4.0-
> 
> The cluster is created and is in a weird state. We should reject these types 
> of repos...
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  60f5ccea67 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ClusterRequest.java
>  aea2072c65 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceResourceProvider.java
>  2b4bf3bba9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
>  27b1654fd3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  d77d13c00b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  fc4fa868b5 
> 
> 
> Diff: https://reviews.apache.org/r/61746/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 61746: Prevent New Clusters from Being Provisioned With PATCH/MAINT Repos

2017-08-21 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61746/#review183326
---


Ship it!




Ship It!

- Robert Levas


On Aug. 18, 2017, 9:23 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61746/
> ---
> 
> (Updated Aug. 18, 2017, 9:23 p.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko, Dmitro Lisnichenko, Nate Cole, 
> and Robert Levas.
> 
> 
> Bugs: AMBARI-21758
> https://issues.apache.org/jira/browse/AMBARI-21758
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> New clusters should only be able to be provisioned using a {{STANDARD}} 
> repository. 
> 
> STR:
> - Register a 2.5.0.0-1234 {{STANDARD}} repo
> - Register a 2.5.4.0- {{MAINT} repo
> - Use a blueprint go create a repo for 2.5.4.0-
> 
> The cluster is created and is in a weird state. We should reject these types 
> of repos...
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  60f5ccea67 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ClusterRequest.java
>  aea2072c65 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceResourceProvider.java
>  2b4bf3bba9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
>  27b1654fd3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java
>  d77d13c00b 
>   
> ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java
>  fc4fa868b5 
> 
> 
> Diff: https://reviews.apache.org/r/61746/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 61707: Pre-configure services when Kerberos is enabled to reduce number of core service restarts when services are added

2017-08-21 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61707/#review183327
---


Ship it!





ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
Line 120 (original), 120-123 (patched)


I know we traditionally put this stuff in AmbariMetaInfo, but we should be 
moving this type of stuff into StackModule/StackDirectory.


- Nate Cole


On Aug. 17, 2017, 8:51 a.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61707/
> ---
> 
> (Updated Aug. 17, 2017, 8:51 a.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene 
> Chekanskiy, Jonathan Hurley, Laszlo Puskas, Nate Cole, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21602
> https://issues.apache.org/jira/browse/AMBARI-21602
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Pre-configure (certain) services when Kerberos is enabled to reduce number of 
> core service restarts when services are added.  
> 
> While processing the Kerberos descriptor, include services marked to be 
> _pre-configured_.  When a tagged service is encountered, process it weather 
> it is installed or not. However if it is not installed, only apply 
> configuration changes for existing configuration types.  This will set at 
> least the core-site changes related to proxyuser and auth-to-local rules 
> properties. By doing this, if a tagged service is later installed, the 
> settings will already be in place in the existing service configs and thus 
> the existing services will not need to be restarted. 
> 
> Caveats:
> - Default values for the uninstalled, tagged, services will be assumed 
> - The stack advisor will be used to suggest locations of components - used to 
> build the clusterHostInfo structure that may be used to derive property 
> values. 
> 
> Note: This processing is to occur when Kerberos is enabled.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  eb97ee376d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/StackAdvisorRequest.java
>  7ba1b1887d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/DeleteIdentityHandler.java
>  3329e76226 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
>  3819863763 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  6c6c43911b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java
>  91a84ea643 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterKerberosDescriptorResourceProvider.java
>  59bd96a8f5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackVersionResourceProvider.java
>  64ead405eb 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/utilities/RemovableIdentities.java
>  d4bb501231 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/utilities/UsedIdentities.java
>  46f5642eb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/AbstractPrepareKerberosServerAction.java
>  dd2b2237e5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/KerberosServerAction.java
>  2e331bb77f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PreconfigureServiceType.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareDisableKerberosServerAction.java
>  60523cdd75 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareEnableKerberosServerAction.java
>  ca15695564 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/PrepareKerberosIdentitiesServerAction.java
>  f239cffd93 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
>  78aaa77a48 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackDirectory.java
>  23fd0a95f5 
>   ambari-server/src/main/java/org/apache/ambari/server/stack/StackModule.java 
> ff1d37808d 
>   ambari-server/src/main/java/org/apache/ambari/server/state/StackInfo.java 
> 149579f57d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/AbstractKerberosDescriptor.java
>  7f53daa357 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/KerberosDescriptor.java
>  eba1b3aecc 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/kerberos/Ke

Review Request 61781: Cache hashes should not be reset after restart ; some ambari-server restart fixes

2017-08-21 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61781/
---

Review request for Ambari, Alejandro Fernandez, Dmytro Sen, and Sid Wagle.


Bugs: AMBARI-21763
https://issues.apache.org/jira/browse/AMBARI-21763


Repository: ambari


Description
---


Diffs
-

  ambari-agent/src/main/python/ambari_agent/AlertStatusReporter.py 20cb717 
  ambari-agent/src/main/python/ambari_agent/ClusterAlertDefinitionsCache.py 
a1f7199 
  ambari-agent/src/main/python/ambari_agent/ClusterCache.py 2316866 
  ambari-agent/src/main/python/ambari_agent/ClusterTopologyCache.py 559a956 
  ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py e7b7e49 
  ambari-agent/src/main/python/ambari_agent/ComponentStatusExecutor.py 66df15a 
  ambari-agent/src/main/python/ambari_agent/HostStatusReporter.py c60ea36 
  ambari-agent/src/main/python/ambari_agent/InitializerModule.py 8208b32 
  
ambari-agent/src/main/python/ambari_agent/listeners/AlertDefinitionsEventListener.py
 cf72a4d 
  
ambari-agent/src/main/python/ambari_agent/listeners/ConfigurationEventListener.py
 e32c503 
  
ambari-agent/src/main/python/ambari_agent/listeners/HostLevelParamsEventListener.py
 aee2992 
  ambari-agent/src/main/python/ambari_agent/listeners/MetadataEventListener.py 
1e9b6e7 
  ambari-agent/src/main/python/ambari_agent/listeners/TopologyEventListener.py 
19a1d32 
  ambari-agent/src/main/python/ambari_agent/security.py a505658 
  ambari-agent/src/test/python/ambari_agent/TestAgentStompResponses.py 55c489f 


Diff: https://reviews.apache.org/r/61781/diff/1/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 61781: Cache hashes should not be reset after restart ; some ambari-server restart fixes

2017-08-21 Thread Andrew Onischuk


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61781/
---

(Updated Aug. 21, 2017, 1:23 p.m.)


Review request for Ambari, Alejandro Fernandez, Dmytro Sen, Myroslav 
Papirkovskyy, and Sid Wagle.


Bugs: AMBARI-21763
https://issues.apache.org/jira/browse/AMBARI-21763


Repository: ambari


Description (updated)
---

.


Diffs
-

  ambari-agent/src/main/python/ambari_agent/AlertStatusReporter.py 20cb717 
  ambari-agent/src/main/python/ambari_agent/ClusterAlertDefinitionsCache.py 
a1f7199 
  ambari-agent/src/main/python/ambari_agent/ClusterCache.py 2316866 
  ambari-agent/src/main/python/ambari_agent/ClusterTopologyCache.py 559a956 
  ambari-agent/src/main/python/ambari_agent/CommandStatusDict.py e7b7e49 
  ambari-agent/src/main/python/ambari_agent/ComponentStatusExecutor.py 66df15a 
  ambari-agent/src/main/python/ambari_agent/HostStatusReporter.py c60ea36 
  ambari-agent/src/main/python/ambari_agent/InitializerModule.py 8208b32 
  
ambari-agent/src/main/python/ambari_agent/listeners/AlertDefinitionsEventListener.py
 cf72a4d 
  
ambari-agent/src/main/python/ambari_agent/listeners/ConfigurationEventListener.py
 e32c503 
  
ambari-agent/src/main/python/ambari_agent/listeners/HostLevelParamsEventListener.py
 aee2992 
  ambari-agent/src/main/python/ambari_agent/listeners/MetadataEventListener.py 
1e9b6e7 
  ambari-agent/src/main/python/ambari_agent/listeners/TopologyEventListener.py 
19a1d32 
  ambari-agent/src/main/python/ambari_agent/security.py a505658 
  ambari-agent/src/test/python/ambari_agent/TestAgentStompResponses.py 55c489f 


Diff: https://reviews.apache.org/r/61781/diff/1/


Testing
---

mvn clean test


Thanks,

Andrew Onischuk

Re: Review Request 61582: Not able to start Yarn services after restoring the configs to initial value

2017-08-21 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61582/
---

(Updated Сер. 21, 2017, 2:18 після полудня)


Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-21173
https://issues.apache.org/jira/browse/AMBARI-21173


Repository: ambari


Description
---

Change Yarn-site.xml to some custom configs and restart Yarn
Restore it back to the origin config; Restart fails
{Code}
Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
 line 106, in 
Nodemanager().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 330, in execute
method(env)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 835, in restart
self.stop(env, upgrade_type=upgrade_type)
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
 line 45, in stop
import params
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params.py",
 line 29, in 
from params_linux import *
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py",
 line 39, in 
import status_params
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/status_params.py",
 line 46, in 
yarn_pid_dir = format("{yarn_pid_dir_prefix}/{yarn_user}")
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
 line 95, in format
return ConfigurationFormatter().format(format_string, args, **result)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
 line 59, in format
result_protected = self.vformat(format_string, args, all_params)
  File "/usr/lib64/python2.7/string.py", line 549, in vformat
result = self._vformat(format_string, args, kwargs, used_args, 2)
  File "/usr/lib64/python2.7/string.py", line 582, in _vformat
result.append(self.format_field(obj, format_spec))
  File "/usr/lib64/python2.7/string.py", line 599, in format_field
return format(value, format_spec)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
 line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in 
configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'yarn-env' 
was not found in configurations dictionary!
{Code}


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 84b411c 


Diff: https://reviews.apache.org/r/61582/diff/2/

Changes: https://reviews.apache.org/r/61582/diff/1-2/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Review Request 61784: (preview) Service and Patch Upgrade Catalog Changes for 2.6 - patch for trunk

2017-08-21 Thread Dmitro Lisnichenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61784/
---

Review request for Ambari, Jonathan Hurley and Nate Cole.


Bugs: AMBARI-21728
https://issues.apache.org/jira/browse/AMBARI-21728


Repository: ambari


Description
---

Service and Patch Upgrade Catalog Changes for 2.6 - patch for trunk


Diffs
-

  ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessor.java 
ef343d5b39 
  ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 
04e4c66759 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/DbmsHelper.java
 b30d01bb3c 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/GenericDbmsHelper.java
 e2a1f38bd2 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/H2Helper.java
 91905e4770 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/MySqlHelper.java
 0daea72b4b 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/OracleHelper.java
 73356d16e3 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/PostgresHelper.java
 37c11840a4 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java
 bcc8328366 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
 2227675478 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
 PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
 230cd950ca 
  
ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
 b4ffbf19c1 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/61784/diff/1/


Testing
---

tests are running

Current upgrade to trunk is broken due to existing code that removes logsearch 
config properties. As a result, new logsearch configs become selected, but have 
no service mapping and DB consistency check fails. Investigating that.


Thanks,

Dmitro Lisnichenko

Re: Review Request 61582: Not able to start Yarn services after restoring the configs to initial value

2017-08-21 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61582/#review183344
---


Ship it!




Can you please add comments about how this fixes the issue?

- Sid Wagle


On Aug. 21, 2017, 2:18 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61582/
> ---
> 
> (Updated Aug. 21, 2017, 2:18 p.m.)
> 
> 
> Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-21173
> https://issues.apache.org/jira/browse/AMBARI-21173
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Change Yarn-site.xml to some custom configs and restart Yarn
> Restore it back to the origin config; Restart fails
> {Code}
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
>  line 106, in 
> Nodemanager().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 330, in execute
> method(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 835, in restart
> self.stop(env, upgrade_type=upgrade_type)
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
>  line 45, in stop
> import params
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params.py",
>  line 29, in 
> from params_linux import *
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py",
>  line 39, in 
> import status_params
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/status_params.py",
>  line 46, in 
> yarn_pid_dir = format("{yarn_pid_dir_prefix}/{yarn_user}")
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
>  line 95, in format
> return ConfigurationFormatter().format(format_string, args, **result)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
>  line 59, in format
> result_protected = self.vformat(format_string, args, all_params)
>   File "/usr/lib64/python2.7/string.py", line 549, in vformat
> result = self._vformat(format_string, args, kwargs, used_args, 2)
>   File "/usr/lib64/python2.7/string.py", line 582, in _vformat
> result.append(self.format_field(obj, format_spec))
>   File "/usr/lib64/python2.7/string.py", line 599, in format_field
> return format(value, format_spec)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
>  line 73, in __getattr__
> raise Fail("Configuration parameter '" + self.name + "' was not found in 
> configurations dictionary!")
> resource_management.core.exceptions.Fail: Configuration parameter 'yarn-env' 
> was not found in configurations dictionary!
> {Code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  84b411c 
> 
> 
> Diff: https://reviews.apache.org/r/61582/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Review Request 61785: When Matching New VDFs for Parent Repos only Consider STANDARD Types

2017-08-21 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61785/
---

Review request for Ambari, Dmytro Grinenko and Nate Cole.


Bugs: AMBARI-21766
https://issues.apache.org/jira/browse/AMBARI-21766


Repository: ambari


Description
---

When registering a {{MAINT}} VDF where the regex matches both an existing 
{{STANDARD}} and existing {{PATCH}} repository, the new VDF registration will 
fail with:
{quote}
{
"status": 400,
"message": "Patch 2.5.4.0-121 was found to match more than one repository 
in use: 2.5.0.0-1237, 2.5.4.0-121. Move all services to a common version and 
try again."
}
{quote}

We should only be matching for parents against the {{STANDARD}} repo type.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
 ea592e50af 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/RepositoryVersionDAO.java
 40bbd072f2 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
 aed0aafbc2 
  
ambari-server/src/test/java/org/apache/ambari/server/orm/dao/RepositoryVersionDAOTest.java
 d41b0ba144 


Diff: https://reviews.apache.org/r/61785/diff/1/


Testing
---

mvn clean test


Thanks,

Jonathan Hurley

Review Request 61786: AMBARI-21768. Spark History Server uses wrong log dir

2017-08-21 Thread Attila Doroszlai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/
---

Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, 
Sid Wagle, and Tim Thorpe.


Bugs: AMBARI-21768
https://issues.apache.org/jira/browse/AMBARI-21768


Repository: ambari


Description
---

* Get rid of unnecessary `-Dspark.history.fs.logDirectory`.  The value is taken 
from `spark-defaults` anyway.
* Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they 
point to the cluster's existing Spark log directory.


Diffs
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
 ad20bf9d9427b6891c4d5944fa3b3e347dbffe12 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
 b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd 


Diff: https://reviews.apache.org/r/61786/diff/1/


Testing
---

Manual test according to steps in the bug.
Tested with both default and customized log directory location.
Verified that both pre-upgrade and post-upgrade jobs are shown in Spark History 
Server UI.


Thanks,

Attila Doroszlai

Re: Review Request 61786: AMBARI-21768. Spark History Server uses wrong log dir

2017-08-21 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183346
---


Ship it!




No 4.2.5 chnaged needed?

- Sid Wagle


On Aug. 21, 2017, 4:23 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> ---
> 
> (Updated Aug. 21, 2017, 4:23 p.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, 
> Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`.  The value is 
> taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they 
> point to the cluster's existing Spark log directory.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  ad20bf9d9427b6891c4d5944fa3b3e347dbffe12 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd 
> 
> 
> Diff: https://reviews.apache.org/r/61786/diff/1/
> 
> 
> Testing
> ---
> 
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark 
> History Server UI.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 61744: Allow for keytab regeneration to be filtered for hosts

2017-08-21 Thread Eugene Chekanskiy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61744/
---

(Updated Aug. 21, 2017, 4:31 p.m.)


Review request for Ambari, Robert Levas and Sebastian Toader.


Bugs: AMBARI-21757
https://issues.apache.org/jira/browse/AMBARI-21757


Repository: ambari


Description
---

regenerateKeytab request now can accept host list and components list to 
regenerate keytabs for.

Example: 
'http://lc6401.ambari.apache.org:8080/api/v1/clusters/cl1?regenerate_keytabs=all&Kerberos/hosts=lc6401.ambari.apache.org&Kerberos/components=HDFS:NAMENODE;DATANODE,YARN:RESOURCEMANAGER'


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/api/resources/ClusterResourceDefinition.java
 f689841 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
 3819863 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 6c6c439 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 b220999 


Diff: https://reviews.apache.org/r/61744/diff/2/

Changes: https://reviews.apache.org/r/61744/diff/1-2/


Testing
---

mvn clean test. manual tests on deployed cluster.


Thanks,

Eugene Chekanskiy

Review Request 61787: zeppelin principal and livy.superusers property do not match on upgraded cluster from Ambari 2.4.2 -and HDP 2.5.5

2017-08-21 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61787/
---

Review request for Ambari, Bikas Saha, Balázs Bence Sári, Eugene Chekanskiy, 
Laszlo Puskas, Sebastian Toader, and Jeff Zhang.


Bugs: AMBARI-21769
https://issues.apache.org/jira/browse/AMBARI-21769


Repository: ambari


Description
---

In a cluster where the Spark and/or Spark2 Livy servers and Zeppelin are 
installed and Kerberos is enabled, it is expected that that 
`livy-conf/livy.superusers` and `livy2-conf/livy.superusers` contain the 
principal name of the Zeppelin user.  However, this value is not always set, 
depending on the order in which the services were installed, when Kerberos was 
enabled, and whether an Ambari or stack upgrade was involved.  And if it is 
set, the value may be incorrect since the Kerberos descriptor assumes the 
Zeppelin principal is `zeppelin-`

The solution is to move the logic to set the `livy-conf/livy.superusers` and 
`livy2-conf/livy.superusers` to the stack advisor to the appropriate value can 
be added as needed.  Also while upgrading to Ambari 2.5.2, the value(s) should 
be fixed if necessary.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/DeconstructedPrincipal.java
 764324bf0c 
  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
 2227675478 
  
ambari-server/src/main/resources/common-services/SPARK/2.2.0/service_advisor.py 
b876cd7714 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.3.0/service_advisor.py
 9ff9b8bc46 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json 
aa81edb2a9 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
246bbcc121 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
872f78b4c7 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/kerberos.json 
0f99bbb634 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 
e9b8d15b5b 
  ambari-server/src/main/resources/stacks/stack_advisor.py 321ac4ec71 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java
 b71b335b95 
  
ambari-server/src/test/python/common-services/SPARK/2.2.0/test_service_advisor.py
 PRE-CREATION 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
51d1678b9f 
  ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py 
63e2229195 


Diff: https://reviews.apache.org/r/61787/diff/1/


Testing
---

Manually tested various scenarios

# Local test results: PENDING

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 61744: Allow for keytab regeneration to be filtered for hosts

2017-08-21 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61744/#review183348
---


Ship it!




Ship It!

- Robert Levas


On Aug. 21, 2017, 12:31 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61744/
> ---
> 
> (Updated Aug. 21, 2017, 12:31 p.m.)
> 
> 
> Review request for Ambari, Robert Levas and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21757
> https://issues.apache.org/jira/browse/AMBARI-21757
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> regenerateKeytab request now can accept host list and components list to 
> regenerate keytabs for.
> 
> Example: 
> 'http://lc6401.ambari.apache.org:8080/api/v1/clusters/cl1?regenerate_keytabs=all&Kerberos/hosts=lc6401.ambari.apache.org&Kerberos/components=HDFS:NAMENODE;DATANODE,YARN:RESOURCEMANAGER'
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/resources/ClusterResourceDefinition.java
>  f689841 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java
>  3819863 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  6c6c439 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  b220999 
> 
> 
> Diff: https://reviews.apache.org/r/61744/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean test. manual tests on deployed cluster.
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Re: Review Request 61786: AMBARI-21768. Spark History Server uses wrong log dir

2017-08-21 Thread Di Li


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183349
---




ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
Lines 379 (patched)


Spark2 config in IOP 4.2.5 also has these properties. They are defined 
under different config types though, i.e spark2-default...


- Di Li


On Aug. 21, 2017, 4:23 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> ---
> 
> (Updated Aug. 21, 2017, 4:23 p.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, 
> Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`.  The value is 
> taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they 
> point to the cluster's existing Spark log directory.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  ad20bf9d9427b6891c4d5944fa3b3e347dbffe12 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd 
> 
> 
> Diff: https://reviews.apache.org/r/61786/diff/1/
> 
> 
> Testing
> ---
> 
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark 
> History Server UI.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 61785: When Matching New VDFs for Parent Repos only Consider STANDARD Types

2017-08-21 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61785/#review183355
---


Ship it!




Ship It!

- Nate Cole


On Aug. 21, 2017, 11:54 a.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61785/
> ---
> 
> (Updated Aug. 21, 2017, 11:54 a.m.)
> 
> 
> Review request for Ambari, Dmytro Grinenko and Nate Cole.
> 
> 
> Bugs: AMBARI-21766
> https://issues.apache.org/jira/browse/AMBARI-21766
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> When registering a {{MAINT}} VDF where the regex matches both an existing 
> {{STANDARD}} and existing {{PATCH}} repository, the new VDF registration will 
> fail with:
> {quote}
> {
> "status": 400,
> "message": "Patch 2.5.4.0-121 was found to match more than one repository 
> in use: 2.5.0.0-1237, 2.5.4.0-121. Move all services to a common version and 
> try again."
> }
> {quote}
> 
> We should only be matching for parents against the {{STANDARD}} repo type.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/VersionDefinitionResourceProvider.java
>  ea592e50af 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/RepositoryVersionDAO.java
>  40bbd072f2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/RepositoryVersionEntity.java
>  aed0aafbc2 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/dao/RepositoryVersionDAOTest.java
>  d41b0ba144 
> 
> 
> Diff: https://reviews.apache.org/r/61785/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 61784: (preview) Service and Patch Upgrade Catalog Changes for 2.6 - patch for trunk

2017-08-21 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61784/#review183356
---


Ship it!




Ship It!

- Nate Cole


On Aug. 21, 2017, 10:35 a.m., Dmitro Lisnichenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61784/
> ---
> 
> (Updated Aug. 21, 2017, 10:35 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley and Nate Cole.
> 
> 
> Bugs: AMBARI-21728
> https://issues.apache.org/jira/browse/AMBARI-21728
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Service and Patch Upgrade Catalog Changes for 2.6 - patch for trunk
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessor.java 
> ef343d5b39 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/DBAccessorImpl.java 
> 04e4c66759 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/DbmsHelper.java
>  b30d01bb3c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/GenericDbmsHelper.java
>  e2a1f38bd2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/H2Helper.java
>  91905e4770 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/MySqlHelper.java
>  0daea72b4b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/OracleHelper.java
>  73356d16e3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/helpers/dbms/PostgresHelper.java
>  37c11840a4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/SchemaUpgradeHelper.java
>  bcc8328366 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
>  2227675478 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog300.java
>  230cd950ca 
>   
> ambari-server/src/test/java/org/apache/ambari/server/orm/DBAccessorImplTest.java
>  b4ffbf19c1 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/61784/diff/1/
> 
> 
> Testing
> ---
> 
> tests are running
> 
> Current upgrade to trunk is broken due to existing code that removes 
> logsearch config properties. As a result, new logsearch configs become 
> selected, but have no service mapping and DB consistency check fails. 
> Investigating that.
> 
> 
> Thanks,
> 
> Dmitro Lisnichenko
> 
>

Re: Review Request 61043: AMBARI-21325: Quicklink support through Knox

2017-08-21 Thread Balázs Bence Sári


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61043/#review183359
---


Ship it!




Ship It!

- Balázs Bence Sári


On Aug. 17, 2017, 9:40 p.m., Chandana Mirashi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61043/
> ---
> 
> (Updated Aug. 17, 2017, 9:40 p.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Juanjo  Marron, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21325
> https://issues.apache.org/jira/browse/AMBARI-21325
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add quicklink support through Knox.
> Changes:
> 1. Add new json properties knox_url, knox_path, supports_knox
>a. knox_url: template to be used for urls that are proxied through Knox 
>b. knox_path: Knox gateway path that will be added to the proxy url.
>c. supports_knox: whether link will be redirected through Knox
> 2. Add above json properties to quicklinks.json
> 3. Add HDFSUI & DATANODE,YARNUI & NODEUI, JOBHISTORYUI, HBASEUI, OOZIEUI, 
> SPARKUI services to Knox topology template.
> 4. Automate protocol and port added to Knox topology file. Based on whether 
> SSL is enabled for the services listed above, the port and protocol in 
> params_linux.py will be updated.
> 5. Update quick_view_link_view.js so that when Knox is installed and 
> support_knox is true, quicklink url follows knox url template specified in 
> the quicklinks.json for the service/component.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/quicklinks/Link.java
>  1d2e712 
>   
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py
>  4558069 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HBASE/quicklinks/quicklinks.json
>  5568122 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/HDFS/quicklinks/quicklinks.json
>  a4216e3 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/OOZIE/quicklinks/quicklinks.json
>  43cf641 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/quicklinks-mapred/quicklinks.json
>  36f71b5 
>   
> ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/quicklinks/quicklinks.json
>  0aca8e3 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/KNOX/configuration/topology.xml
>  df4c1b4 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/quicklinks/quicklinks.json
>  PRE-CREATION 
>   ambari-server/src/test/python/stacks/2.2/configs/knox_upgrade.json 1805c3b 
>   ambari-web/app/views/common/quick_view_link_view.js 5888acb 
> 
> 
> Diff: https://reviews.apache.org/r/61043/diff/3/
> 
> 
> Testing
> ---
> 
> 1. ambari-server: mvn test
> 2. mvn -DskipSurefireTests -Dpython.test.mask=test_knox_gateway.py test
> 3. ambari-web: mvn test
>  21213 passing (54s)
>  128 pending
> 
> 
> Thanks,
> 
> Chandana Mirashi
> 
>

Re: Review Request 61786: AMBARI-21768. Spark History Server uses wrong log dir

2017-08-21 Thread Bikas Saha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183361
---




ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
Lines 380 (patched)


This does not seem right because the log dir seems tied to a specific 
version. 4.2.0.0.


- Bikas Saha


On Aug. 21, 2017, 9:23 a.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> ---
> 
> (Updated Aug. 21, 2017, 9:23 a.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, 
> Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`.  The value is 
> taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they 
> point to the cluster's existing Spark log directory.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  ad20bf9d9427b6891c4d5944fa3b3e347dbffe12 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd 
> 
> 
> Diff: https://reviews.apache.org/r/61786/diff/1/
> 
> 
> Testing
> ---
> 
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark 
> History Server UI.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 61787: zeppelin principal and livy.superusers property do not match on upgraded cluster from Ambari 2.4.2 -and HDP 2.5.5

2017-08-21 Thread Nate Cole


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61787/#review183373
---


Ship it!





ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
Lines 187-189 (patched)


How could this be null?



ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
Lines 224 (patched)


nit: MapUtils.isEmpty(map)


- Nate Cole


On Aug. 21, 2017, 12:34 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61787/
> ---
> 
> (Updated Aug. 21, 2017, 12:34 p.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Balázs Bence Sári, Eugene Chekanskiy, 
> Laszlo Puskas, Sebastian Toader, and Jeff Zhang.
> 
> 
> Bugs: AMBARI-21769
> https://issues.apache.org/jira/browse/AMBARI-21769
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In a cluster where the Spark and/or Spark2 Livy servers and Zeppelin are 
> installed and Kerberos is enabled, it is expected that that 
> `livy-conf/livy.superusers` and `livy2-conf/livy.superusers` contain the 
> principal name of the Zeppelin user.  However, this value is not always set, 
> depending on the order in which the services were installed, when Kerberos 
> was enabled, and whether an Ambari or stack upgrade was involved.  And if it 
> is set, the value may be incorrect since the Kerberos descriptor assumes the 
> Zeppelin principal is `zeppelin-`
> 
> The solution is to move the logic to set the `livy-conf/livy.superusers` and 
> `livy2-conf/livy.superusers` to the stack advisor to the appropriate value 
> can be added as needed.  Also while upgrading to Ambari 2.5.2, the value(s) 
> should be fixed if necessary.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/DeconstructedPrincipal.java
>  764324bf0c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
>  2227675478 
>   
> ambari-server/src/main/resources/common-services/SPARK/2.2.0/service_advisor.py
>  b876cd7714 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.3.0/service_advisor.py
>  9ff9b8bc46 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json 
> aa81edb2a9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 246bbcc121 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> 872f78b4c7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/kerberos.json 
> 0f99bbb634 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 
> e9b8d15b5b 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 321ac4ec71 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java
>  b71b335b95 
>   
> ambari-server/src/test/python/common-services/SPARK/2.2.0/test_service_advisor.py
>  PRE-CREATION 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 51d1678b9f 
>   ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py 
> 63e2229195 
> 
> 
> Diff: https://reviews.apache.org/r/61787/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested various scenarios
> 
> # Local test results: PENDING
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 61787: zeppelin principal and livy.superusers property do not match on upgraded cluster from Ambari 2.4.2 -and HDP 2.5.5

2017-08-21 Thread Robert Levas



> On Aug. 21, 2017, 3:48 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
> > Lines 187-189 (patched)
> > 
> >
> > How could this be null?

You are correct... looking at the _current_ implemention of 
KerberosDescriptorFactory it wouldn't be null.  I sometimes get paranoid about 
`null`s.


> On Aug. 21, 2017, 3:48 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
> > Lines 224 (patched)
> > 
> >
> > nit: MapUtils.isEmpty(map)

I should get used to using the *Utils classes.  If I make any changes, I will 
look into that.


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61787/#review183373
---


On Aug. 21, 2017, 12:34 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61787/
> ---
> 
> (Updated Aug. 21, 2017, 12:34 p.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Balázs Bence Sári, Eugene Chekanskiy, 
> Laszlo Puskas, Sebastian Toader, and Jeff Zhang.
> 
> 
> Bugs: AMBARI-21769
> https://issues.apache.org/jira/browse/AMBARI-21769
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In a cluster where the Spark and/or Spark2 Livy servers and Zeppelin are 
> installed and Kerberos is enabled, it is expected that that 
> `livy-conf/livy.superusers` and `livy2-conf/livy.superusers` contain the 
> principal name of the Zeppelin user.  However, this value is not always set, 
> depending on the order in which the services were installed, when Kerberos 
> was enabled, and whether an Ambari or stack upgrade was involved.  And if it 
> is set, the value may be incorrect since the Kerberos descriptor assumes the 
> Zeppelin principal is `zeppelin-`
> 
> The solution is to move the logic to set the `livy-conf/livy.superusers` and 
> `livy2-conf/livy.superusers` to the stack advisor to the appropriate value 
> can be added as needed.  Also while upgrading to Ambari 2.5.2, the value(s) 
> should be fixed if necessary.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/DeconstructedPrincipal.java
>  764324bf0c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
>  2227675478 
>   
> ambari-server/src/main/resources/common-services/SPARK/2.2.0/service_advisor.py
>  b876cd7714 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.3.0/service_advisor.py
>  9ff9b8bc46 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json 
> aa81edb2a9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 246bbcc121 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> 872f78b4c7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/kerberos.json 
> 0f99bbb634 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 
> e9b8d15b5b 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 321ac4ec71 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java
>  b71b335b95 
>   
> ambari-server/src/test/python/common-services/SPARK/2.2.0/test_service_advisor.py
>  PRE-CREATION 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 51d1678b9f 
>   ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py 
> 63e2229195 
> 
> 
> Diff: https://reviews.apache.org/r/61787/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested various scenarios
> 
> # Local test results: PENDING
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 61582: Not able to start Yarn services after restoring the configs to initial value

2017-08-21 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61582/
---

(Updated Сер. 21, 2017, 8:01 після полудня)


Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-21173
https://issues.apache.org/jira/browse/AMBARI-21173


Repository: ambari


Description
---

Change Yarn-site.xml to some custom configs and restart Yarn
Restore it back to the origin config; Restart fails
{Code}
Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
 line 106, in 
Nodemanager().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 330, in execute
method(env)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 835, in restart
self.stop(env, upgrade_type=upgrade_type)
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
 line 45, in stop
import params
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params.py",
 line 29, in 
from params_linux import *
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py",
 line 39, in 
import status_params
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/status_params.py",
 line 46, in 
yarn_pid_dir = format("{yarn_pid_dir_prefix}/{yarn_user}")
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
 line 95, in format
return ConfigurationFormatter().format(format_string, args, **result)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
 line 59, in format
result_protected = self.vformat(format_string, args, all_params)
  File "/usr/lib64/python2.7/string.py", line 549, in vformat
result = self._vformat(format_string, args, kwargs, used_args, 2)
  File "/usr/lib64/python2.7/string.py", line 582, in _vformat
result.append(self.format_field(obj, format_spec))
  File "/usr/lib64/python2.7/string.py", line 599, in format_field
return format(value, format_spec)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
 line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in 
configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'yarn-env' 
was not found in configurations dictionary!
{Code}


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 84b411c 


Diff: https://reviews.apache.org/r/61582/diff/3/

Changes: https://reviews.apache.org/r/61582/diff/2-3/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 61786: AMBARI-21768. Spark History Server uses wrong log dir

2017-08-21 Thread Attila Doroszlai



> On Aug. 21, 2017, 7:09 p.m., Di Li wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
> > Lines 379 (patched)
> > 
> >
> > Spark2 config in IOP 4.2.5 also has these properties. They are defined 
> > under different config types though, i.e spark2-default...

Upgrade from 4.2.5 works fine. IOP 4.2.5's default values for both 
`spark.eventLog.dir` and `spark.history.fs.logDirectory` match those of HDP 
(`hdfs:///spark2-history/`), thus the problem does not exist there.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183349
---


On Aug. 21, 2017, 6:23 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> ---
> 
> (Updated Aug. 21, 2017, 6:23 p.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, 
> Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`.  The value is 
> taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they 
> point to the cluster's existing Spark log directory.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  ad20bf9d9427b6891c4d5944fa3b3e347dbffe12 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd 
> 
> 
> Diff: https://reviews.apache.org/r/61786/diff/1/
> 
> 
> Testing
> ---
> 
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark 
> History Server UI.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 61786: AMBARI-21768. Spark History Server uses wrong log dir

2017-08-21 Thread Attila Doroszlai



> On Aug. 21, 2017, 8:28 p.m., Bikas Saha wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
> > Lines 380 (patched)
> > 
> >
> > This does not seem right because the log dir seems tied to a specific 
> > version. 4.2.0.0.

The version number does not matter, could be anything as long as it's the same 
as before the upgrade.  This is the default value in IOP 4.2, and this 
replacement only applies for the non-customized case.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183361
---


On Aug. 21, 2017, 6:23 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> ---
> 
> (Updated Aug. 21, 2017, 6:23 p.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, 
> Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`.  The value is 
> taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they 
> point to the cluster's existing Spark log directory.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  ad20bf9d9427b6891c4d5944fa3b3e347dbffe12 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd 
> 
> 
> Diff: https://reviews.apache.org/r/61786/diff/1/
> 
> 
> Testing
> ---
> 
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark 
> History Server UI.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Review Request 61792: Spark1 Shuffle Property Is Removed Incorrectly on a Stack Upgrade

2017-08-21 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61792/
---

Review request for Ambari, Nate Cole and Robert Levas.


Bugs: AMBARI-21770
https://issues.apache.org/jira/browse/AMBARI-21770


Repository: ambari


Description
---

You can read the description at 
https://issues.apache.org/jira/browse/AMBARI-21770 for more detail. Long story 
short is that the `spark_shuffle` and `spark2_shuffle` properties are 
difference in BigInsights. Where a default install of Spark1 in BI 4.2.0 
doesn't even add `spark_shuffle`, the Spark2 install in BI 4.2.5 adds it 
instead of `spark2_shuffle`.

There was a Jira (https://issues.apache.org/jira/browse/AMBARI-21421) which 
removed `spark_shuffle` via a find replace so that HDP's `spark2_shuffle` would 
take over. But this led to problems after adding spark post-upgrade.

The cleanest solution seemed to be to just reset the properties to the expected 
defaults if Spark was installed. I used the spark-env/content key to determine 
if Spark was installed.


Diffs
-

  
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
 f55f9fb1a9 
  
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
 ad20bf9d94 


Diff: https://reviews.apache.org/r/61792/diff/1/


Testing
---

IOP 4.2 to HDP 2.6 upgrade


Thanks,

Jonathan Hurley

Re: Review Request 61582: Not able to start Yarn services after restoring the configs to initial value

2017-08-21 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61582/#review183381
---


Ship it!




Ship It!

- Sid Wagle


On Aug. 21, 2017, 8:01 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61582/
> ---
> 
> (Updated Aug. 21, 2017, 8:01 p.m.)
> 
> 
> Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-21173
> https://issues.apache.org/jira/browse/AMBARI-21173
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Change Yarn-site.xml to some custom configs and restart Yarn
> Restore it back to the origin config; Restart fails
> {Code}
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
>  line 106, in 
> Nodemanager().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 330, in execute
> method(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 835, in restart
> self.stop(env, upgrade_type=upgrade_type)
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
>  line 45, in stop
> import params
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params.py",
>  line 29, in 
> from params_linux import *
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py",
>  line 39, in 
> import status_params
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/status_params.py",
>  line 46, in 
> yarn_pid_dir = format("{yarn_pid_dir_prefix}/{yarn_user}")
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
>  line 95, in format
> return ConfigurationFormatter().format(format_string, args, **result)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
>  line 59, in format
> result_protected = self.vformat(format_string, args, all_params)
>   File "/usr/lib64/python2.7/string.py", line 549, in vformat
> result = self._vformat(format_string, args, kwargs, used_args, 2)
>   File "/usr/lib64/python2.7/string.py", line 582, in _vformat
> result.append(self.format_field(obj, format_spec))
>   File "/usr/lib64/python2.7/string.py", line 599, in format_field
> return format(value, format_spec)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
>  line 73, in __getattr__
> raise Fail("Configuration parameter '" + self.name + "' was not found in 
> configurations dictionary!")
> resource_management.core.exceptions.Fail: Configuration parameter 'yarn-env' 
> was not found in configurations dictionary!
> {Code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  84b411c 
> 
> 
> Diff: https://reviews.apache.org/r/61582/diff/3/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 61582: Not able to start Yarn services after restoring the configs to initial value

2017-08-21 Thread Sid Wagle


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61582/#review183382
---




ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
Lines 1800 (patched)


Fix internal id number.


- Sid Wagle


On Aug. 21, 2017, 8:01 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61582/
> ---
> 
> (Updated Aug. 21, 2017, 8:01 p.m.)
> 
> 
> Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-21173
> https://issues.apache.org/jira/browse/AMBARI-21173
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Change Yarn-site.xml to some custom configs and restart Yarn
> Restore it back to the origin config; Restart fails
> {Code}
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
>  line 106, in 
> Nodemanager().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 330, in execute
> method(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 835, in restart
> self.stop(env, upgrade_type=upgrade_type)
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
>  line 45, in stop
> import params
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params.py",
>  line 29, in 
> from params_linux import *
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py",
>  line 39, in 
> import status_params
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/status_params.py",
>  line 46, in 
> yarn_pid_dir = format("{yarn_pid_dir_prefix}/{yarn_user}")
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
>  line 95, in format
> return ConfigurationFormatter().format(format_string, args, **result)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
>  line 59, in format
> result_protected = self.vformat(format_string, args, all_params)
>   File "/usr/lib64/python2.7/string.py", line 549, in vformat
> result = self._vformat(format_string, args, kwargs, used_args, 2)
>   File "/usr/lib64/python2.7/string.py", line 582, in _vformat
> result.append(self.format_field(obj, format_spec))
>   File "/usr/lib64/python2.7/string.py", line 599, in format_field
> return format(value, format_spec)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
>  line 73, in __getattr__
> raise Fail("Configuration parameter '" + self.name + "' was not found in 
> configurations dictionary!")
> resource_management.core.exceptions.Fail: Configuration parameter 'yarn-env' 
> was not found in configurations dictionary!
> {Code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  84b411c 
> 
> 
> Diff: https://reviews.apache.org/r/61582/diff/3/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 61792: Spark1 Shuffle Property Is Removed Incorrectly on a Stack Upgrade

2017-08-21 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61792/#review183384
---


Ship it!




Ship It!

- Robert Levas


On Aug. 21, 2017, 4:20 p.m., Jonathan Hurley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61792/
> ---
> 
> (Updated Aug. 21, 2017, 4:20 p.m.)
> 
> 
> Review request for Ambari, Nate Cole and Robert Levas.
> 
> 
> Bugs: AMBARI-21770
> https://issues.apache.org/jira/browse/AMBARI-21770
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> You can read the description at 
> https://issues.apache.org/jira/browse/AMBARI-21770 for more detail. Long 
> story short is that the `spark_shuffle` and `spark2_shuffle` properties are 
> difference in BigInsights. Where a default install of Spark1 in BI 4.2.0 
> doesn't even add `spark_shuffle`, the Spark2 install in BI 4.2.5 adds it 
> instead of `spark2_shuffle`.
> 
> There was a Jira (https://issues.apache.org/jira/browse/AMBARI-21421) which 
> removed `spark_shuffle` via a find replace so that HDP's `spark2_shuffle` 
> would take over. But this led to problems after adding spark post-upgrade.
> 
> The cleanest solution seemed to be to just reset the properties to the 
> expected defaults if Spark was installed. I used the spark-env/content key to 
> determine if Spark was installed.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml
>  f55f9fb1a9 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  ad20bf9d94 
> 
> 
> Diff: https://reviews.apache.org/r/61792/diff/1/
> 
> 
> Testing
> ---
> 
> IOP 4.2 to HDP 2.6 upgrade
> 
> 
> Thanks,
> 
> Jonathan Hurley
> 
>

Re: Review Request 61582: Not able to start Yarn services after restoring the configs to initial value

2017-08-21 Thread Vitalyi Brodetskyi


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61582/
---

(Updated Сер. 21, 2017, 8:48 після полудня)


Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid Wagle.


Bugs: AMBARI-21173
https://issues.apache.org/jira/browse/AMBARI-21173


Repository: ambari


Description
---

Change Yarn-site.xml to some custom configs and restart Yarn
Restore it back to the origin config; Restart fails
{Code}
Traceback (most recent call last):
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
 line 106, in 
Nodemanager().execute()
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 330, in execute
method(env)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
 line 835, in restart
self.stop(env, upgrade_type=upgrade_type)
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
 line 45, in stop
import params
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params.py",
 line 29, in 
from params_linux import *
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py",
 line 39, in 
import status_params
  File 
"/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/status_params.py",
 line 46, in 
yarn_pid_dir = format("{yarn_pid_dir_prefix}/{yarn_user}")
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
 line 95, in format
return ConfigurationFormatter().format(format_string, args, **result)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
 line 59, in format
result_protected = self.vformat(format_string, args, all_params)
  File "/usr/lib64/python2.7/string.py", line 549, in vformat
result = self._vformat(format_string, args, kwargs, used_args, 2)
  File "/usr/lib64/python2.7/string.py", line 582, in _vformat
result.append(self.format_field(obj, format_spec))
  File "/usr/lib64/python2.7/string.py", line 599, in format_field
return format(value, format_spec)
  File 
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
 line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in 
configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'yarn-env' 
was not found in configurations dictionary!
{Code}


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
 84b411c 


Diff: https://reviews.apache.org/r/61582/diff/4/

Changes: https://reviews.apache.org/r/61582/diff/3-4/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 61582: Not able to start Yarn services after restoring the configs to initial value

2017-08-21 Thread Myroslav Papirkovskyy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61582/#review183386
---


Ship it!




Ship It!

- Myroslav Papirkovskyy


On Сер. 21, 2017, 11:48 після полудня, Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61582/
> ---
> 
> (Updated Сер. 21, 2017, 11:48 після полудня)
> 
> 
> Review request for Ambari, Myroslav Papirkovskyy, Sumit Mohanty, and Sid 
> Wagle.
> 
> 
> Bugs: AMBARI-21173
> https://issues.apache.org/jira/browse/AMBARI-21173
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Change Yarn-site.xml to some custom configs and restart Yarn
> Restore it back to the origin config; Restart fails
> {Code}
> Traceback (most recent call last):
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
>  line 106, in 
> Nodemanager().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 330, in execute
> method(env)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
>  line 835, in restart
> self.stop(env, upgrade_type=upgrade_type)
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/nodemanager.py",
>  line 45, in stop
> import params
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params.py",
>  line 29, in 
> from params_linux import *
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py",
>  line 39, in 
> import status_params
>   File 
> "/var/lib/ambari-agent/cache/common-services/YARN/3.0.0.3.0/package/scripts/status_params.py",
>  line 46, in 
> yarn_pid_dir = format("{yarn_pid_dir_prefix}/{yarn_user}")
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
>  line 95, in format
> return ConfigurationFormatter().format(format_string, args, **result)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py",
>  line 59, in format
> result_protected = self.vformat(format_string, args, all_params)
>   File "/usr/lib64/python2.7/string.py", line 549, in vformat
> result = self._vformat(format_string, args, kwargs, used_args, 2)
>   File "/usr/lib64/python2.7/string.py", line 582, in _vformat
> result.append(self.format_field(obj, format_spec))
>   File "/usr/lib64/python2.7/string.py", line 599, in format_field
> return format(value, format_spec)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py",
>  line 73, in __getattr__
> raise Fail("Configuration parameter '" + self.name + "' was not found in 
> configurations dictionary!")
> resource_management.core.exceptions.Fail: Configuration parameter 'yarn-env' 
> was not found in configurations dictionary!
> {Code}
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java
>  84b411c 
> 
> 
> Diff: https://reviews.apache.org/r/61582/diff/4/
> 
> 
> Testing
> ---
> 
> mvn clean test
> 
> 
> Thanks,
> 
> Vitalyi Brodetskyi
> 
>

Re: Review Request 61787: zeppelin principal and livy.superusers property do not match on upgraded cluster from Ambari 2.4.2 -and HDP 2.5.5

2017-08-21 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61787/#review183387
---


Ship it!




Ship It!

- Jonathan Hurley


On Aug. 21, 2017, 12:34 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61787/
> ---
> 
> (Updated Aug. 21, 2017, 12:34 p.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Balázs Bence Sári, Eugene Chekanskiy, 
> Laszlo Puskas, Sebastian Toader, and Jeff Zhang.
> 
> 
> Bugs: AMBARI-21769
> https://issues.apache.org/jira/browse/AMBARI-21769
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> In a cluster where the Spark and/or Spark2 Livy servers and Zeppelin are 
> installed and Kerberos is enabled, it is expected that that 
> `livy-conf/livy.superusers` and `livy2-conf/livy.superusers` contain the 
> principal name of the Zeppelin user.  However, this value is not always set, 
> depending on the order in which the services were installed, when Kerberos 
> was enabled, and whether an Ambari or stack upgrade was involved.  And if it 
> is set, the value may be incorrect since the Kerberos descriptor assumes the 
> Zeppelin principal is `zeppelin-`
> 
> The solution is to move the logic to set the `livy-conf/livy.superusers` and 
> `livy2-conf/livy.superusers` to the stack advisor to the appropriate value 
> can be added as needed.  Also while upgrading to Ambari 2.5.2, the value(s) 
> should be fixed if necessary.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/kerberos/DeconstructedPrincipal.java
>  764324bf0c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
>  2227675478 
>   
> ambari-server/src/main/resources/common-services/SPARK/2.2.0/service_advisor.py
>  b876cd7714 
>   
> ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.3.0/service_advisor.py
>  9ff9b8bc46 
>   
> ambari-server/src/main/resources/stacks/HDP/2.5/services/SPARK/kerberos.json 
> aa81edb2a9 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 246bbcc121 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/kerberos.json 
> 872f78b4c7 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/kerberos.json 
> 0f99bbb634 
>   ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 
> e9b8d15b5b 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 321ac4ec71 
>   
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java
>  b71b335b95 
>   
> ambari-server/src/test/python/common-services/SPARK/2.2.0/test_service_advisor.py
>  PRE-CREATION 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 51d1678b9f 
>   ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py 
> 63e2229195 
> 
> 
> Diff: https://reviews.apache.org/r/61787/diff/1/
> 
> 
> Testing
> ---
> 
> Manually tested various scenarios
> 
> # Local test results: PENDING
> 
> # Jenkins test results: PENDING
> 
> 
> Thanks,
> 
> Robert Levas
> 
>

Re: Review Request 61786: AMBARI-21768. Spark History Server uses wrong log dir

2017-08-21 Thread Jonathan Hurley


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183388
---




ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
Lines 380-381 (patched)


I thought that we went through properties like this and changed them to 
"hdp" ... Wouldn't this cause a problem with that replacement?


- Jonathan Hurley


On Aug. 21, 2017, 12:23 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> ---
> 
> (Updated Aug. 21, 2017, 12:23 p.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, 
> Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`.  The value is 
> taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they 
> point to the cluster's existing Spark log directory.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  ad20bf9d9427b6891c4d5944fa3b3e347dbffe12 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd 
> 
> 
> Diff: https://reviews.apache.org/r/61786/diff/1/
> 
> 
> Testing
> ---
> 
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark 
> History Server UI.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Review Request 61803: zeppelin proxy user settings are not configured in core-site.xml on upgraded cluster from Ambari 2.4.2

2017-08-21 Thread Robert Levas


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61803/
---

Review request for Ambari, Balázs Bence Sári, Eugene Chekanskiy, Jonathan 
Hurley, Laszlo Puskas, Prabhjyot Singh, and Sebastian Toader.


Bugs: AMBARI-21772
https://issues.apache.org/jira/browse/AMBARI-21772


Repository: ambari


Description
---

The configurations `core-site/hadoop.proxyuser.zeppelin.hosts` and 
`core-site/hadoop.proxyuser.zeppelin.groups`, which are required to allow 
zeppelin user to proxy other users in order to support impersonation features 
of few of the zeppelin interpreters, was not added during the upgrade from 
Ambari 2.4.2 to Ambari 2.5.2. Impersonation support for Zeppelin was added in 
AMBARI-20094.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog252.java
 1192c112f8 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/kerberos.json
 925215b8c5 
  
ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.3.0/kerberos.json
 925215b8c5 
  
ambari-server/src/main/resources/stacks/HDP/2.5/services/ZEPPELIN/kerberos.json 
925215b8c5 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/ZEPPELIN/kerberos.json 
925215b8c5 
  
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog252Test.java
 d7df68c122 


Diff: https://reviews.apache.org/r/61803/diff/1/


Testing
---

Manually tested various scenarios

# Local test results: PENDING

# Jenkins test results: PENDING


Thanks,

Robert Levas

Re: Review Request 61786: AMBARI-21768. Spark History Server uses wrong log dir

2017-08-21 Thread Attila Doroszlai



> On Aug. 21, 2017, 10:59 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
> > Lines 380-381 (patched)
> > 
> >
> > I thought that we went through properties like this and changed them to 
> > "hdp" ... Wouldn't this cause a problem with that replacement?

"Change stack references" happens before "Upgrade service configs", so the 
"iop" introduced here is not subject to replacement.


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61786/#review183388
---


On Aug. 21, 2017, 6:23 p.m., Attila Doroszlai wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61786/
> ---
> 
> (Updated Aug. 21, 2017, 6:23 p.m.)
> 
> 
> Review request for Ambari, Bikas Saha, Di Li, Jonathan Hurley, Sumit Mohanty, 
> Sid Wagle, and Tim Thorpe.
> 
> 
> Bugs: AMBARI-21768
> https://issues.apache.org/jira/browse/AMBARI-21768
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> * Get rid of unnecessary `-Dspark.history.fs.logDirectory`.  The value is 
> taken from `spark-defaults` anyway.
> * Tweak `spark.eventLog.dir` and `spark.history.fs.logDirectory` so that they 
> point to the cluster's existing Spark log directory.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
>  ad20bf9d9427b6891c4d5944fa3b3e347dbffe12 
>   
> ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
>  b8c23bbb5292d374d4fb53b9b43a97ebcb1853cd 
> 
> 
> Diff: https://reviews.apache.org/r/61786/diff/1/
> 
> 
> Testing
> ---
> 
> Manual test according to steps in the bug.
> Tested with both default and customized log directory location.
> Verified that both pre-upgrade and post-upgrade jobs are shown in Spark 
> History Server UI.
> 
> 
> Thanks,
> 
> Attila Doroszlai
> 
>

Re: Review Request 61709: Ambari should expose whether a service is able to rolling-restart instead of hard coding service name

2017-08-21 Thread Jayush Luniya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61709/#review183449
---


Ship it!




Ship It!

- Jayush Luniya


On Aug. 17, 2017, 7:22 a.m., Madhuvanthi Radhakrishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61709/
> ---
> 
> (Updated Aug. 17, 2017, 7:22 a.m.)
> 
> 
> Review request for Ambari, Jaimin Jetly, Jayush Luniya, Koji Sekiguchi, and 
> Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21737
> https://issues.apache.org/jira/browse/AMBARI-21737
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add a new property called rollingRestartAllowed to metainfo.xml Existing 
> logic of client components are not removed.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceComponentResponse.java
>  0b0a5da2b0 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceComponentResourceProvider.java
>  764e394540 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/ComponentInfo.java 
> a8a9a0faef 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/metainfo.xml
>  f408ba38ac 
>   ambari-server/src/main/resources/common-services/KAFKA/0.8.1/metainfo.xml 
> d322adc47b 
>   ambari-server/src/main/resources/properties.json 11ca7f678a 
>   ambari-web/app/mappers/stack_service_mapper.js 8931066f76 
>   ambari-web/app/models/stack_service_component.js eb6f2dbff7 
> 
> 
> Diff: https://reviews.apache.org/r/61709/diff/1/
> 
> 
> Testing
> ---
> 
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> [INFO] Total time: 25:12 min
> [INFO] Finished at: 2017-08-17T00:01:59-07:00
> [INFO] Final Memory: 85M/962M
> [INFO] 
> 
> 
> Install ambari
> Deploy HDP
> Ensure no regression
> Kafka brokers allow rolling restart
> 
> This property can be used for any other services which have components that 
> are not clients but support rolling restart.
> 
> 
> Thanks,
> 
> Madhuvanthi Radhakrishnan
> 
>

43 matches

Mail list logo