Review Request 64852: Ambari Infra Manager: deleting documents
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64852/ --- Review request for Ambari, Attila Magyar, Oliver Szabo, and Sid Wagle. Bugs: AMBARI-22653 and AMBARI-22702 https://issues.apache.org/jira/browse/AMBARI-22653 https://issues.apache.org/jira/browse/AMBARI-22702 Repository: ambari Description --- * defining new job for deleting documents from solr: collection and filter field can be defined in properties * archiving job supports deleting the succesfully uploaded documents * date time format in output filename can be defined in property file * s3 secret and access keys can be defined in a separate file and this file can be referenced from infra-manager.properties * generic property processing for jobs * Helper class for processing parametrized solr query text * new IT case: Running archiving job with a bigger start value than end value exports and deletes 0 documents * new IT case: After Deleting job deletes documents from solr no document found in the specified interval * existing IT cases for archiving also check that solr has no more documents left in the specified interval after archiving job finished Diffs - ambari-infra/ambari-infra-assembly/pom.xml bf0e6791d4 ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java 9db562cec3 ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java 703e1cfa67 ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java 4a09d7d7ee ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story cd1f49d052 ambari-infra/ambari-infra-manager/build.xml 3d0f4da8a0 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/JobProperties.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/PropertyMap.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/SolrDAOBase.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java 7a30393224 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java 8421802cd7 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java 1f3957a034 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java 189591102b ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java 3b6c402458 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java 37f6d1b669 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java 9af4afc630 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentWiper.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java d3f2a65e07 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileNameSuffixFormatter.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/ItemWriterListener.java 742777109e ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java baad61bdcb ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalItemWriterListener.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java 0979f10bfc ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java deeb9c7ea0 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDAO.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java 2e7341d365 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java 5ded9acf85 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrParametrizedString.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrProperties.java PRE-CREATION ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java b3ea14ed65 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java 55ba58a0f2
Re: Review Request 64836: AMBARI-22696 Whitelist execute latency from Storm Ambari metrics (trunk)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64836/#review194562 --- Ship it! Ship It! - Aravindan Vijayan On Dec. 25, 2017, 4:36 a.m., Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64836/ > --- > > (Updated Dec. 25, 2017, 4:36 a.m.) > > > Review request for Ambari, Arun Mahadevan and Aravindan Vijayan. > > > Bugs: AMBARI-22696 > https://issues.apache.org/jira/browse/AMBARI-22696 > > > Repository: ambari > > > Description > --- > > We have an use case which leverages execute latency along with process > latency. Given that we only register process latency as whitelist, we would > need to add execute latency as whitelist as well. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/configuration/storm-site.xml > 6b97fb6e41 > > ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/service_advisor.py > 1d6bbe0f9f > > ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml > 558beaf8d9 > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 6f409cdfd1 > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > cc6bd9dca4 > > > Diff: https://reviews.apache.org/r/64836/diff/1/ > > > Testing > --- > > Manually applied the change to configuration of test cluster, and confirmed > it works. > > It doesn't mean I have tested it with upgrading, so please consider verifying > it works well with various upgrade cases. > > > Thanks, > > Jungtaek Lim > >
Re: Review Request 64837: AMBARI-22696 Whitelist execute latency from Storm Ambari metrics (branch-2.6)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64837/#review194561 --- Ship it! Ship It! - Aravindan Vijayan On Dec. 25, 2017, 4:38 a.m., Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64837/ > --- > > (Updated Dec. 25, 2017, 4:38 a.m.) > > > Review request for Ambari, Arun Mahadevan and Aravindan Vijayan. > > > Bugs: AMBARI-22696 > https://issues.apache.org/jira/browse/AMBARI-22696 > > > Repository: ambari > > > Description > --- > > We have an use case which leverages execute latency along with process > latency. Given that we only register process latency as whitelist, we would > need to add execute latency as whitelist as well. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml > 558beaf8d9 > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 4187dc9b55 > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > 8f27d79375 > > > Diff: https://reviews.apache.org/r/64837/diff/1/ > > > Testing > --- > > Manually applied the change to configuration of test cluster, and confirmed > it works. > > It doesn't mean I have tested it with upgrading, so please consider verifying > it works well with various upgrade cases. > > > Thanks, > > Jungtaek Lim > >
Re: Review Request 64837: AMBARI-22696 Whitelist execute latency from Storm Ambari metrics (branch-2.6)
> On 12월 28, 2017, 1:31 오전, Aravindan Vijayan wrote: > > These changes might be needed in > > ambari/ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/service_advisor.py > > as well. That file only exists on trunk branch. I've also published another review board for trunk branch. - Jungtaek --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64837/#review194559 --- On 12월 25, 2017, 4:38 오전, Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64837/ > --- > > (Updated 12월 25, 2017, 4:38 오전) > > > Review request for Ambari, Arun Mahadevan and Aravindan Vijayan. > > > Bugs: AMBARI-22696 > https://issues.apache.org/jira/browse/AMBARI-22696 > > > Repository: ambari > > > Description > --- > > We have an use case which leverages execute latency along with process > latency. Given that we only register process latency as whitelist, we would > need to add execute latency as whitelist as well. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml > 558beaf8d9 > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 4187dc9b55 > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > 8f27d79375 > > > Diff: https://reviews.apache.org/r/64837/diff/1/ > > > Testing > --- > > Manually applied the change to configuration of test cluster, and confirmed > it works. > > It doesn't mean I have tested it with upgrading, so please consider verifying > it works well with various upgrade cases. > > > Thanks, > > Jungtaek Lim > >
Re: Review Request 64837: AMBARI-22696 Whitelist execute latency from Storm Ambari metrics (branch-2.6)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64837/#review194559 --- These changes might be needed in ambari/ambari-server/src/main/resources/common-services/STORM/1.0.1.3.0/service_advisor.py as well. - Aravindan Vijayan On Dec. 25, 2017, 4:38 a.m., Jungtaek Lim wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64837/ > --- > > (Updated Dec. 25, 2017, 4:38 a.m.) > > > Review request for Ambari, Arun Mahadevan and Aravindan Vijayan. > > > Bugs: AMBARI-22696 > https://issues.apache.org/jira/browse/AMBARI-22696 > > > Repository: ambari > > > Description > --- > > We have an use case which leverages execute latency along with process > latency. Given that we only register process latency as whitelist, we would > need to add execute latency as whitelist as well. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-site.xml > 558beaf8d9 > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 4187dc9b55 > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > 8f27d79375 > > > Diff: https://reviews.apache.org/r/64837/diff/1/ > > > Testing > --- > > Manually applied the change to configuration of test cluster, and confirmed > it works. > > It doesn't mean I have tested it with upgrading, so please consider verifying > it works well with various upgrade cases. > > > Thanks, > > Jungtaek Lim > >
Review Request 64853: UpgradeUserKerberosDescriptor is not executed during stack upgrade due to missing target stack data
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64853/
---
Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene Chekanskiy,
Jonathan Hurley, Nate Cole, Robert Nettleton, and Sandor Molnar.
Bugs: AMBARI-22693
https://issues.apache.org/jira/browse/AMBARI-22693
Repository: ambari
Description
---
UpgradeUserKerberosDescriptor is not executed during stack upgrade due to
missing target stack data.
*Steps to reproduce*
# Deploy cluster with Ambari version 2.6.0 and HDP version 2.4
** Storm should be installed to guarantee an error
# Do Express upgrade to HDP version 2.6
# Regenerate Keytabs.
Upon restarting Storm the following error is encountered
```
Exception in thread "main" java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at clojure.lang.RT.classForName(RT.java:2154)
at clojure.lang.RT.classForName(RT.java:2163)
at clojure.lang.RT.loadClassForName(RT.java:2182)
at clojure.lang.RT.load(RT.java:436)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5671)
at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
at clojure.core$load_lib.doInvoke(core.clj:5710)
at clojure.lang.RestFn.applyTo(RestFn.java:142)
at clojure.core$apply.invoke(core.clj:632)
at clojure.core$load_libs.doInvoke(core.clj:5749)
at clojure.lang.RestFn.applyTo(RestFn.java:137)
at clojure.core$apply.invoke(core.clj:632)
at clojure.core$require.doInvoke(core.clj:5832)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at
org.apache.storm.daemon.nimbus$loading__5340__auto982.invoke(nimbus.clj:16)
at org.apache.storm.daemon.nimbus__init.load(Unknown Source)
at org.apache.storm.daemon.nimbus__init.(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at clojure.lang.RT.classForName(RT.java:2154)
at clojure.lang.RT.classForName(RT.java:2163)
at clojure.lang.RT.loadClassForName(RT.java:2182)
at clojure.lang.RT.load(RT.java:436)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.lang.Var.invoke(Var.java:379)
at org.apache.storm.daemon.nimbus.(Unknown Source)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
backtype.storm.security.auth.KerberosPrincipalToLocal
at
org.apache.storm.security.auth.AuthUtils.GetPrincipalToLocalPlugin(AuthUtils.java:125)
at
org.apache.storm.security.auth.authorizer.ImpersonationAuthorizer.prepare(ImpersonationAuthorizer.java:54)
at
org.apache.storm.daemon.common$mk_authorization_handler.invoke(common.clj:417)
at org.apache.storm.ui.core__init.load(Unknown Source)
at org.apache.storm.ui.core__init.(Unknown Source)
... 35 more
Caused by: java.lang.ClassNotFoundException:
backtype.storm.security.auth.KerberosPrincipalToLocal
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.storm.security.auth.AuthUtils.GetPrincipalToLocalPlugin(AuthUtils.java:121)
... 39 more
```
#Cause
In the following code snip, {{targetStackID}} is {null}}:
*`org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java:103`*:
```
StackId targetStackId =
getStackIdFromCommandParams(KeyNames.TARGET_STACK);
```
This causes the logic in `UpgradeUserKerberosDescriptor` to be skipped.
#Solution
Change the code snip from above to
*`org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java:103`*
```
StackId targetStackId = cluster.getDesiredStackVersion();
```
NOTE: This has already been fixed in the trunk (Ambari 3.0.0).
Diffs
-
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
ffe2ed63d1
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptor.java
a8c254a3eb
ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/UpgradeUserKerberosDescriptorTest.java
69d82b2ab4
Re: Review Request 64850: Fixing interface and file renaming issues
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64850/ --- (Updated Dec. 27, 2017, 10:20 a.m.) Review request for Ambari, Attila Doroszlai, Oliver Szabo, and Sid Wagle. Bugs: AMBARI-22653 https://issues.apache.org/jira/browse/AMBARI-22653 Repository: ambari Description (updated) --- Patch https://reviews.apache.org/r/64607 contains two renamed interfaces: DocumentIterator -> CloseableIterator DocumentSource -> ItemWriterListener but the files containing these interface definitions were not renamed on trunk git repo Using java.util.ArrayList insted of com.google.common.collect.Lists.newArrayList() Diffs (updated) - ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraManagerStories.java cf720ef0d0 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java 5fa29b0099 ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java 742777109e Diff: https://reviews.apache.org/r/64850/diff/2/ Changes: https://reviews.apache.org/r/64850/diff/1-2/ Testing --- maven build unit and integration tests Thanks, Krisztian Kasa
Re: Review Request 64850: Fixing interface and file renaming issues
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64850/#review194527 --- Ship it! Ship It! - Oliver Szabo On Dec. 27, 2017, 8:10 a.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/64850/ > --- > > (Updated Dec. 27, 2017, 8:10 a.m.) > > > Review request for Ambari, Attila Doroszlai, Oliver Szabo, and Sid Wagle. > > > Bugs: AMBARI-22653 > https://issues.apache.org/jira/browse/AMBARI-22653 > > > Repository: ambari > > > Description > --- > > Patch https://reviews.apache.org/r/64607 contains two renamed interfaces: > DocumentIterator -> CloseableIterator > DocumentSource -> ItemWriterListener > > but the files containing these interface definitions were not renamed on > trunk git repo > > > Diffs > - > > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java > > > ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java > > > > Diff: https://reviews.apache.org/r/64850/diff/1/ > > > Testing > --- > > maven build > unit and integration tests > > > Thanks, > > Krisztian Kasa > >
Review Request 64850: Fixing interface and file renaming issues
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64850/ --- Review request for Ambari, Attila Doroszlai, Oliver Szabo, and Sid Wagle. Bugs: AMBARI-22653 https://issues.apache.org/jira/browse/AMBARI-22653 Repository: ambari Description --- Patch https://reviews.apache.org/r/64607 contains two renamed interfaces: DocumentIterator -> CloseableIterator DocumentSource -> ItemWriterListener but the files containing these interface definitions were not renamed on trunk git repo Diffs - ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java Diff: https://reviews.apache.org/r/64850/diff/1/ Testing --- maven build unit and integration tests Thanks, Krisztian Kasa
