[jira] [Commented] (AMBARI-22514) Infra Manager: Schedulable document deletion & archiving for Infra Solr
[ https://issues.apache.org/jira/browse/AMBARI-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300600#comment-16300600 ] Hudson commented on AMBARI-22514: - FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #8545 (See [https://builds.apache.org/job/Ambari-trunk-Commit/8545/]) AMBARI-22514, AMBARI-22653. Ambari Infra Manager: solr data exporting (swagle: [http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=36d0271f74a70f5cfeca0e5ca0ebeb795fab6138]) * (edit) ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java * (add) ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraManagerStories.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java * (add) ambari-infra/ambari-infra-manager-it/src/test/resources/stories/infra_api_tests.story * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java * (edit) ambari-infra/pom.xml * (add) ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/ExportJobsSteps.java * (add) ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/OffsetDateTimeConverter.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java * (edit) ambari-infra/ambari-infra-manager/docker/Dockerfile * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportPropertyMap.java * (add) ambari-infra/ambari-infra-manager-it/pom.xml * (edit) ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java * (edit) ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java * (add) ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/InfraClient.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java * (add) ambari-infra/ambari-infra-manager/docker/docker-compose.yml * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java * (add) ambari-infra/ambari-infra-manager-it/src/test/java/org/apache/ambari/infra/steps/AbstractInfraSteps.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/ObjectSource.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java * (delete) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java * (add) ambari-infra/ambari-infra-manager-it/src/test/resources/log4j.properties * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java * (add) ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExportPropertiesTest.java * (edit) ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java * (add) ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryPropertiesTest.java * (add) ambari-infra/ambari-infra-manager/docker/infra-manager-docker-compose.sh * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/AbstractFileAction.java > Infra Manager: Schedulable document deletion & archiving for Infra Solr > --- > > Key: AMBARI-22514 > URL: https://issues.apache.org/jira/browse/
[jira] [Commented] (AMBARI-22514) Infra Manager: Schedulable document deletion & archiving for Infra Solr
[ https://issues.apache.org/jira/browse/AMBARI-22514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16268953#comment-16268953 ] Hudson commented on AMBARI-22514: - SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #8452 (See [https://builds.apache.org/job/Ambari-trunk-Commit/8452/]) AMBARI-22514. Initial implementation of Schedulable document deletion & (oleewere: [http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=393fdb8048ff579e8a55cd1b477a23d1bf105576]) * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentIterator.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemWriter.java * (edit) ambari-infra/ambari-infra-manager/src/main/resources/log4j2.xml * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/FileAction.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentDestination.java * (add) ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentExporterTest.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportProperties.java * (edit) ambari-logsearch/docker/test-config/logfeeder/logfeeder.properties * (add) ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/DocumentItemReaderTest.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentSource.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriter.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrDocumentIterator.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/CompositeFileAction.java * (edit) ambari-infra/ambari-infra-manager/pom.xml * (add) ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/SolrQueryBuilderTest.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportConfiguration.java * (add) ambari-infra/ambari-infra-manager/src/test/java/org/apache/ambari/infra/job/archive/LocalDocumentItemWriterTest.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportStepListener.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Uploader.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/TarGzCompressor.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryProperties.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentItemReader.java * (edit) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/manager/JobManager.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExportJobListener.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentExporter.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/S3Properties.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/DocumentSource.java * (edit) ambari-infra/ambari-infra-manager/src/main/resources/infra-manager.properties * (add) ambari-infra/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/Document.java > Infra Manager: Schedulable document deletion & archiving for Infra Solr > --- > > Key: AMBARI-22514 > URL: https://issues.apache.org/jira/browse/AMBARI-22514 > Project: Ambari > Issue Type: Bug > Components: ambari-infra >Affects Versions: 3.0.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Critical > Fix For: 3.0.0 > > Original Estimate: 240h > Remaining Estimate: 240h > > If the data size is too big, and the max retention is not good enough, we > should delete the latest logs from solr, but of course we need the data > itself, so we should archive the data somewhere else (like s3 or hdfs) > This feature should also provide support for configurable policies that users > can configure in order to determine how long data is held and/or archived. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
