[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713178#comment-16713178 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #77 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/77/]) AMBARI-24833. Get rid of custom time based triggering policy (oleewere: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=26485f24e5f72f1d1c801f72bc718927c3daed12]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CustomTimeBasedTriggeringPolicy.java * (edit) ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712165#comment-16712165 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #72 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/72/]) AMBARI-24833. Cloud base path can be URI as well (oleewere: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=a739360576790a52f8f67969a3ce97f08874d13e]) * (edit) docs/api-docs/logsearch-swagger.yaml * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/resources/core-site.xml * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java * (edit) docs/logfeeder_properties.md * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java * (add) ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploaderTest.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703606#comment-16703606 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #61 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/61/]) AMBARI-24833. ABFS (ADLS v2) fs implementation was not included on the (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=e186a93689cbb0d8c78e74227378c598eba94091]) * (edit) pom.xml * (add) ambari-logsearch-logfeeder/src/main/resources/core-site-examples/s3-core-site.xml * (add) ambari-logsearch-logfeeder/src/main/resources/core-site-examples/wasb-core-site.xml * (add) ambari-logsearch-logfeeder/src/main/resources/core-site-examples/abfs-core-site.xml > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703595#comment-16703595
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #49: AMBARI-24833. ABFS (ADLS v2) fs
implementation was not included on the classpath
URL: https://github.com/apache/ambari-logsearch/pull/49
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/resources/core-site-examples/abfs-core-site.xml
b/ambari-logsearch-logfeeder/src/main/resources/core-site-examples/abfs-core-site.xml
new file mode 100644
index 00..abbf6a3671
--- /dev/null
+++
b/ambari-logsearch-logfeeder/src/main/resources/core-site-examples/abfs-core-site.xml
@@ -0,0 +1,54 @@
+
+
+
+fs.defaultFS
+abfs://logsea...@myaccount.dfs.core.windows.net
+
+
+fs.azure.account.key.myaccount.dfs.core.windows.net
+mySecretAccessKey
+
+
+fs.adlsGen2.impl
+org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
+
+
+fs.abfss.impl
+org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem
+
+
+fs.AbstractFileSystem.adlsGen2.impl
+org.apache.hadoop.fs.azurebfs.Abfs
+
+
+fs.AbstractFileSystem.abfss.impl
+org.apache.hadoop.fs.azurebfs.Abfss
+
+
+fs.azure.check.block.md5
+false
+
+
+fs.azure.store.blob.md5
+false
+
+
+fs.azure.createRemoteFileSystemDuringInitialization
+true
+
+
\ No newline at end of file
diff --git
a/ambari-logsearch-logfeeder/src/main/resources/core-site-examples/s3-core-site.xml
b/ambari-logsearch-logfeeder/src/main/resources/core-site-examples/s3-core-site.xml
new file mode 100644
index 00..5807a54871
--- /dev/null
+++
b/ambari-logsearch-logfeeder/src/main/resources/core-site-examples/s3-core-site.xml
@@ -0,0 +1,30 @@
+
+
+
+fs.defaultFS
+s3a://logsearch
+
+
+fs.s3a.access.key
+MyAccessKey
+
+
+fs.s3a.secret.key
+MySecretKey
+
+
\ No newline at end of file
diff --git
a/ambari-logsearch-logfeeder/src/main/resources/core-site-examples/wasb-core-site.xml
b/ambari-logsearch-logfeeder/src/main/resources/core-site-examples/wasb-core-site.xml
new file mode 100644
index 00..50f7becf6d
--- /dev/null
+++
b/ambari-logsearch-logfeeder/src/main/resources/core-site-examples/wasb-core-site.xml
@@ -0,0 +1,42 @@
+
+
+
+fs.defaultFS
+wasb://logsea...@myaccount.blob.core.windows.net
+
+
+fs.azure.account.key.myaccount.blob.core.windows.net
+mySecretAccessKey
+
+
+fs.AbstractFileSystem.wasb.impl
+org.apache.hadoop.fs.azure.Wasbs
+
+
+fs.AbstractFileSystem.wasbs.impl
+org.apache.hadoop.fs.azure.Wasbs
+
+
+fs.azure.selfthrottling.read.factor
+1.0
+
+
+fs.azure.selfthrottling.write.factor
+1.0
+
+
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
index 91e5b09cae..53d99b5ceb 100644
--- a/pom.xml
+++ b/pom.xml
@@ -84,7 +84,7 @@
amd64
${deb.python.ver}
7.5.0
-3.1.1
+3.1.1.3.0.2.0-50
2.5
3.4.6.2.3.0.0-2557
4
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703577#comment-16703577 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #49: AMBARI-24833. ABFS (ADLS v2) fs implementation was not included on the classpath URL: https://github.com/apache/ambari-logsearch/pull/49#issuecomment-442932203 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/93/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703559#comment-16703559
]
ASF GitHub Bot commented on AMBARI-24833:
-
swagle commented on a change in pull request #49: AMBARI-24833. ABFS (ADLS v2)
fs implementation was not included on the classpath
URL: https://github.com/apache/ambari-logsearch/pull/49#discussion_r237591557
##
File path: ambari-logsearch-logfeeder/pom.xml
##
@@ -175,12 +175,12 @@
org.apache.hadoop
hadoop-azure
- ${hadoop.version}
+ ${hdp-hadoop.version}
Review comment:
Why this is vendor specific?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703420#comment-16703420 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #49: AMBARI-24833. ABFS (ADLS v2) fs implementation was not included on the classpath URL: https://github.com/apache/ambari-logsearch/pull/49#issuecomment-442891022 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/92/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703429#comment-16703429
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #49: AMBARI-24833. ABFS (ADLS
v2) fs implementation was not included on the classpath
URL: https://github.com/apache/ambari-logsearch/pull/49#discussion_r237552684
##
File path: pom.xml
##
@@ -85,6 +85,7 @@
${deb.python.ver}
7.5.0
3.1.1
+3.1.1.3.0.2.0-50
Review comment:
@adoroszlai that is a released version from HDP repos. if 3.2.0 will go out
to apache release, that will be used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703416#comment-16703416
]
ASF GitHub Bot commented on AMBARI-24833:
-
adoroszlai commented on a change in pull request #49: AMBARI-24833. ABFS (ADLS
v2) fs implementation was not included on the classpath
URL: https://github.com/apache/ambari-logsearch/pull/49#discussion_r237550629
##
File path: pom.xml
##
@@ -85,6 +85,7 @@
${deb.python.ver}
7.5.0
3.1.1
+3.1.1.3.0.2.0-50
Review comment:
Will this need to be updated periodically like some metrics dependency
versions?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703408#comment-16703408 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #49: AMBARI-24833. ABFS (ADLS v2) fs implementation was not included on the classpath URL: https://github.com/apache/ambari-logsearch/pull/49 # What changes were proposed in this pull request? Use specific release of hadoop libraries as ABFS is not included in hadoop 3.1.1 version yet also add some example core-site.xml s i## How was this patch tested? manually with created WASB and ABFS(S) storage accounts on Azure This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694566#comment-16694566 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #41 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/41/]) AMBARI-24833. Do not open FS on main thread. (#33) (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=4e3940a647f170dd6fd9f9c3027ce301dde5dee4]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java * (edit) ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageOutput.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694563#comment-16694563
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #33: AMBARI-24833. Do not open FS on main thread.
URL: https://github.com/apache/ambari-logsearch/pull/33
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
b/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
index 12af637d8c..3f40e88a15 100644
---
a/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
+++
b/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
@@ -111,7 +111,7 @@ private void tryLoadingInputConfig(InputConfigMonitor
inputConfigMonitor, JsonPa
} catch (Exception e) {
final String errorMessage;
if (tries < 3) {
- errorMessage = String.format("Cannot parse input config: %s, will
retry in a few seconds again (tries: %s)", inputConfig, String.valueOf(tries));
+ errorMessage = String.format("Cannot parse input config: '%s', will
retry in a few seconds again (tries: %s)", inputConfig, String.valueOf(tries));
logger.error(errorMessage, e);
try {
Thread.sleep(2000);
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
index a15ac7468a..e5a6e3800b 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
@@ -108,6 +108,7 @@
public static final String CLOUD_STORAGE_MODE =
"logfeeder.cloud.storage.mode";
public static final String CLOUD_STORAGE_DESTINATION =
"logfeeder.cloud.storage.destination";
public static final String CLOUD_STORAGE_UPLOAD_ON_SHUTDOWN =
"logfeeder.cloud.storage.upload.on.shutdown";
+ public static final String CLOUD_STORAGE_UPLOADER_TIMEOUT_MINUTUES =
"logfeeder.cloud.storage.uploader.timeout.minutes";
public static final String CLOUD_STORAGE_UPLOADER_INTERVAL_SECONDS =
"logfeeder.cloud.storage.uploader.interval.seconds";
public static final String CLOUD_STORAGE_BUCKET =
"logfeeder.cloud.storage.bucket";
public static final String CLOUD_STORAGE_BUCKET_BOOTSTRAP =
"logfeeder.cloud.storage.bucket.bootstrap";
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
index b6ab4c7342..9ed4c9b949 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
@@ -251,6 +251,16 @@
@Value("${" + LogFeederConstants.CLOUD_STORAGE_UPLOADER_INTERVAL_SECONDS +
":60}")
private Integer cloudStorageUploaderIntervalSeconds;
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.CLOUD_STORAGE_UPLOADER_TIMEOUT_MINUTUES,
+description = "Timeout value for uploading task to cloud storage in
minutes.",
+examples = {"10"},
+defaultValue = "60",
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${" + LogFeederConstants.CLOUD_STORAGE_UPLOADER_TIMEOUT_MINUTUES +
":60}")
+ private Integer cloudStorageUploaderTimeoutMinutes;
+
@LogSearchPropertyDescription(
name = LogFeederConstants.CLOUD_STORAGE_USE_HDFS_CLIENT,
description = "Use hdfs client with cloud connectors instead of the core
clients for shipping data to cloud storage",
@@ -499,6 +509,14 @@ public void setCloudStorageUploaderIntervalSeconds(Integer
cloudStorageUploaderI
this.cloudStorageUploaderIntervalSeconds =
cloudStorageUploaderIntervalSeconds;
}
+ public Integer getCloudStorageUploaderTimeoutMinutes() {
+return cloudStorageUploaderTimeoutMinutes;
+ }
+
+ public void setCloudStorageUploaderTimeoutMinutes(Integer
cloudStorageUploaderTimeoutMinutes) {
+this.cloudStorageUploaderTimeoutMinutes =
cloudStorageUploaderTimeoutMinutes;
+ }
+
public boolean isUseCloudHdfsClient() {
return useCloudHdfsClient;
}
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfe
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693949#comment-16693949 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #33: AMBARI-24833. Do not open FS on main thread. URL: https://github.com/apache/ambari-logsearch/pull/33#issuecomment-440480912 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/68/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693940#comment-16693940 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #33: AMBARI-24833. Do not open FS on main thread. URL: https://github.com/apache/ambari-logsearch/pull/33#issuecomment-440479766 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/67/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693771#comment-16693771 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #33: AMBARI-24833. Do not open FS on main thread. URL: https://github.com/apache/ambari-logsearch/pull/33#issuecomment-440424235 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/66/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693757#comment-16693757
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #33: AMBARI-24833. Do not open
FS on main thread.
URL: https://github.com/apache/ambari-logsearch/pull/33#discussion_r235158237
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
##
@@ -85,17 +84,16 @@ public void init(LogFeederProps logFeederProps) {
}
logger.info("HDFS client - will use '{}' permission for uploaded files",
hdfsOutputConfig.getHdfsFilePermissions());
LogFeederHDFSUtil.overrideFileSystemConfigs(logFeederProps, configuration);
-this.fs = LogFeederHDFSUtil.buildFileSystem(configuration);
}
@Override
public void upload(String source, String target) throws Exception {
+final FileSystem fs = LogFeederHDFSUtil.buildFileSystem(configuration);
Review comment:
could be but if the filesystem is not accessible it blocks the thread, so it
would not be required, but it would be nice if we could call that on the actual
used object during shutdown
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693740#comment-16693740
]
ASF GitHub Bot commented on AMBARI-24833:
-
swagle commented on a change in pull request #33: AMBARI-24833. Do not open FS
on main thread.
URL: https://github.com/apache/ambari-logsearch/pull/33#discussion_r235153497
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
##
@@ -85,17 +84,16 @@ public void init(LogFeederProps logFeederProps) {
}
logger.info("HDFS client - will use '{}' permission for uploaded files",
hdfsOutputConfig.getHdfsFilePermissions());
LogFeederHDFSUtil.overrideFileSystemConfigs(logFeederProps, configuration);
-this.fs = LogFeederHDFSUtil.buildFileSystem(configuration);
}
@Override
public void upload(String source, String target) throws Exception {
+final FileSystem fs = LogFeederHDFSUtil.buildFileSystem(configuration);
Review comment:
Should this be a try..finally block with close?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693438#comment-16693438 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #33: AMBARI-24833. Do not open FS on main thread. URL: https://github.com/apache/ambari-logsearch/pull/33#issuecomment-440333560 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/65/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693437#comment-16693437 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #33: AMBARI-24833. Do not open FS on main thread. URL: https://github.com/apache/ambari-logsearch/pull/33#issuecomment-440332634 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/64/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693427#comment-16693427 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #33: AMBARI-24833. Do not open FS on main thread. URL: https://github.com/apache/ambari-logsearch/pull/33 # What changes were proposed in this pull request? Do not create FS on main thread, on hybrid mode, it can block the main thread so the solr output will stop working. Note: that is just a temporal solution, FS creation thread should have a timeout, (that fs stuff cannot be interrupted !!! - so need to figure out something else) ## How was this patch tested? stopped localstack/hdfs in hybrid mode Please review @g-boros This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692272#comment-16692272 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #38 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/38/]) AMBARI-24833. Let logfeeder.properties override core-site configs. (#30) (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=defc854b8d1948745a07a1a6e07322670ffef340]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692258#comment-16692258 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #37 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/37/]) AMBARI-24833. Use clustername and hostname in cloud log archive dir (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=1af5b0b50442df0bb292151554a169e64372289a]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692251#comment-16692251
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #29: AMBARI-24833. Use clustername and hostname in
cloud log archive dir
URL: https://github.com/apache/ambari-logsearch/pull/29
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
index 0cfdbcc1e2..ea422b9c31 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
@@ -21,6 +21,7 @@
import org.apache.ambari.logfeeder.common.LogFeederConstants;
import org.apache.ambari.logfeeder.conf.LogFeederProps;
import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.ambari.logfeeder.util.LogFeederUtil;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.core.LoggerContext;
@@ -56,10 +57,11 @@ public static Logger createLogger(Input input,
LoggerContext loggerContext, LogF
String type = input.getLogType().replace(LogFeederConstants.CLOUD_PREFIX,
"");
String uniqueThreadName = input.getThread().getName();
Configuration config = loggerContext.getConfiguration();
-String destination = logFeederProps.getCloudStorageDestination().getText();
String baseDir =
logFeederProps.getRolloverConfig().getRolloverArchiveBaseDir();
-String activeLogDir = Paths.get(baseDir, destination, ACTIVE_FOLDER,
type).toFile().getAbsolutePath();
-String archiveLogDir = Paths.get(baseDir, destination, ARCHIVED_FOLDER,
type).toFile().getAbsolutePath();
+String destination = logFeederProps.getCloudStorageDestination().getText();
+String clusterHostnameBaseDir = Paths.get(baseDir, destination,
logFeederProps.getClusterName(),
LogFeederUtil.hostName).toFile().getAbsolutePath();
+String activeLogDir = Paths.get(clusterHostnameBaseDir, ACTIVE_FOLDER,
type).toFile().getAbsolutePath();
+String archiveLogDir = Paths.get(clusterHostnameBaseDir, ARCHIVED_FOLDER,
type).toFile().getAbsolutePath();
boolean useGzip = logFeederProps.getRolloverConfig().isUseGzip();
final String archiveFilePattern;
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
index af9326aed4..22c7fc1c61 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
@@ -26,6 +26,7 @@
import org.apache.logging.log4j.Logger;
import java.io.File;
+import java.nio.file.Paths;
import java.util.Collection;
/**
@@ -74,10 +75,11 @@ public void run() {
*/
void doUpload() {
try {
- final String archiveLogDir = String.join(File.separator,
logFeederProps.getRolloverConfig().getRolloverArchiveBaseDir(), uploaderType,
"archived");
- if (new File(archiveLogDir).exists()) {
+ final File archiveLogDir =
Paths.get(logFeederProps.getRolloverConfig().getRolloverArchiveBaseDir(),
+uploaderType, clusterName, hostName, "archived").toFile();
+ if (archiveLogDir.exists()) {
String[] extensions = {"log", "json", "gz"};
-Collection filesToUpload = FileUtils.listFiles(new
File(archiveLogDir), extensions, true);
+Collection filesToUpload = FileUtils.listFiles(archiveLogDir,
extensions, true);
if (filesToUpload.isEmpty()) {
logger.debug("Not found any files to upload.");
} else {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Comp
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692252#comment-16692252
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #30: AMBARI-24833. Let logfeeder.properties
override core-site configs.
URL: https://github.com/apache/ambari-logsearch/pull/30
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java
index 5405a1051c..df1b8a55b2 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java
@@ -57,6 +57,7 @@ public void init(LogFeederProps logFeederProps) {
conf.set("fs.s3a.endpoint", s3OutputConfig.getEndpoint());
conf.set("fs.s3a.path.style.access",
String.valueOf(s3OutputConfig.isPathStyleAccess()));
conf.set("fs.s3a.multiobjectdelete.enable",
String.valueOf(s3OutputConfig.isMultiobjectDeleteEnable()));
+LogFeederHDFSUtil.overrideFileSystemConfigs(logFeederProps, conf);
this.fs = LogFeederHDFSUtil.buildFileSystem(conf);
}
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
index 421c4c5cb7..7e1b471de0 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
@@ -84,6 +84,7 @@ public void init(LogFeederProps logFeederProps) {
}
}
logger.info("HDFS client - will use '{}' permission for uploaded files",
hdfsOutputConfig.getHdfsFilePermissions());
+LogFeederHDFSUtil.overrideFileSystemConfigs(logFeederProps, configuration);
this.fs = LogFeederHDFSUtil.buildFileSystem(configuration);
}
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java
index 3549e04af6..b741229a13 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java
@@ -19,7 +19,10 @@
package org.apache.ambari.logfeeder.util;
import java.io.IOException;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.ambari.logfeeder.conf.LogFeederProps;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
@@ -95,4 +98,20 @@ public static void closeFileSystem(FileSystem fileSystem) {
}
}
}
+
+ /**
+ * Override Hadoop configuration object based on logfeeder.properties
configurations (with keys that starts with "fs." or "hadoop.*")
+ * @param logFeederProps global property holder
+ * @param configuration hadoop configuration holder
+ */
+ public static void overrideFileSystemConfigs(LogFeederProps logFeederProps,
Configuration configuration) {
+Properties properties = logFeederProps.getProperties();
+for (Map.Entry prop : properties.entrySet()) {
+ String propertyName = prop.getKey().toString();
+ if (propertyName.startsWith("fs.")) {
+logger.info("Override {} configuration (by logfeeder.properties)",
propertyName);
+configuration.set(propertyName, prop.getValue().toString());
+ }
+}
+ }
}
\ No newline at end of file
diff --git a/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
b/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
index 45c05f39cf..2537913bbe 100644
--- a/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
+++ b/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
@@ -68,3 +68,5 @@ logfeeder.s3.access.key=MyAccessKey
logfeeder.s3.object.acl=public-read
logfeeder.s3.path.style.access=true
logfeeder.s3.multiobject.delete.enable=false
+
+fs.s3a.acl.default=PublicReadWrite
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692249#comment-16692249 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #30: AMBARI-24833. Let logfeeder.properties override core-site configs. URL: https://github.com/apache/ambari-logsearch/pull/30#issuecomment-440036535 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/61/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692235#comment-16692235 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #30: AMBARI-24833. Let logfeeder.properties override core-site configs. URL: https://github.com/apache/ambari-logsearch/pull/30 # What changes were proposed in this pull request? core site configs can be override by logfeeder.properties. It can be useful if no core-site.xml available on a host ## How was this patch tested? FT: manually This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691894#comment-16691894 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #29: AMBARI-24833. Use clustername and hostname in cloud log archive dir URL: https://github.com/apache/ambari-logsearch/pull/29 # What changes were proposed in this pull request? Use hostname + cluster name as well in archive dir (to make it work properly with mounts) ## How was this patch tested? FT manually with docker env please review @g-boros This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691899#comment-16691899 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #29: AMBARI-24833. Use clustername and hostname in cloud log archive dir URL: https://github.com/apache/ambari-logsearch/pull/29#issuecomment-439942525 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/60/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691756#comment-16691756
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #28: AMBARI-24833. Use seconds instead of minutes
for cloud log threshold
URL: https://github.com/apache/ambari-logsearch/pull/28
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
index a15ac7468a..1b0aa534d6 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
@@ -117,7 +117,7 @@
public static final String CLOUD_STORAGE_BASE_PATH =
"logfeeder.cloud.storage.base.path";
public static final String CLOUD_ROLLOVER_ARCHIVE_LOCATION =
"logfeeder.cloud.rollover.archive.base.dir";
- public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_MIN =
"logfeeder.cloud.rollover.threshold.min";
+ public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_SECONDS =
"logfeeder.cloud.rollover.threshold.min";
public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE =
"logfeeder.cloud.rollover.threshold.size";
public static final String CLOUD_ROLLOVER_MAX_BACKUP_FILES =
"logfeeder.cloud.rollover.max.files";
public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE_UNIT =
"logfeeder.cloud.rollover.threshold.size.unit";
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java
index 282792ae86..bdc908bd5e 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java
@@ -37,14 +37,14 @@
private String rolloverArchiveBaseDir;
@LogSearchPropertyDescription(
-name = LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_MIN,
-description = "Rollover cloud log files after an interval (minutes)",
-examples = {"1"},
-defaultValue = "60",
+name = LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SECONDS,
+description = "Rollover cloud log files after an interval (seconds)",
+examples = {"60"},
+defaultValue = "3600",
sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
)
- @Value("${"+ LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_MIN + ":60}")
- private int rolloverThresholdTimeMins;
+ @Value("${"+ LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SECONDS +
":3600}")
+ private int rolloverThresholdTimeSeconds;
@LogSearchPropertyDescription(
name = LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE,
@@ -116,12 +116,12 @@
@Value("${"+ LogFeederConstants.CLOUD_ROLLOVER_ON_STARTUP + ":false}")
private boolean rolloverOnStartup;
- public int getRolloverThresholdTimeMins() {
-return rolloverThresholdTimeMins;
+ public int getRolloverThresholdTimeSeconds() {
+return rolloverThresholdTimeSeconds;
}
- public void setRolloverThresholdTimeMins(int rolloverThresholdTimeMins) {
-this.rolloverThresholdTimeMins = rolloverThresholdTimeMins;
+ public void setRolloverThresholdTimeSeconds(int
rolloverThresholdTimeSeconds) {
+this.rolloverThresholdTimeSeconds = rolloverThresholdTimeSeconds;
}
public Integer getRolloverMaxBackupFiles() {
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
index 0cfdbcc1e2..d39349f4d1 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
@@ -78,7 +78,7 @@ public static Logger createLogger(Input input, LoggerContext
loggerContext, LogF
String rolloverSize =
logFeederProps.getRolloverConfig().getRolloverSize().toString() +
logFeederProps.getRolloverConfig().getRolloverSizeFormat();
SizeBasedTriggeringPolicy sizeBasedTriggeringPolicy =
SizeBasedTriggeringPolicy.createPolicy(rolloverSize);
CustomTimeBasedTriggeringPolicy customTimeBasedTriggeringPolicy =
CustomTimeBasedTriggeringPolicy
-
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691755#comment-16691755 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere commented on issue #28: AMBARI-24833. Use seconds instead of minutes for cloud log threshold URL: https://github.com/apache/ambari-logsearch/pull/28#issuecomment-439908578 i will close this, let's keep as minutes This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691711#comment-16691711 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #28: AMBARI-24833. Use seconds instead of minutes for cloud log threshold URL: https://github.com/apache/ambari-logsearch/pull/28#issuecomment-439897680 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/59/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691700#comment-16691700 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #28: AMBARI-24833. Use seconds instead of minutes for cloud log threshold URL: https://github.com/apache/ambari-logsearch/pull/28 # What changes were proposed in this pull request? Use seconds instead of minutes for cloud log threshold ## How was this patch tested? waiting for UTs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691685#comment-16691685 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #27: AMBARI-24833. HDFS client kerberos support + small fixes URL: https://github.com/apache/ambari-logsearch/pull/27#issuecomment-43945 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/58/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691688#comment-16691688 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #36 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/36/]) AMBARI-24833. HDFS client kerberos support + small fixes (#27) (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=c94d3b20576a179f7a4e04491e423c77a3976ce5]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/CloudStorageInputConfigHandler.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputLineEnricher.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/ExternalHdfsOutputConfig.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/DefaultInputConfigHandler.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/ExternalHDFSUploadClient.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/HdfsOutputConfig.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/UploadClientFactory.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/AbstractInputConfigHandler.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691686#comment-16691686
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #27: AMBARI-24833. HDFS client kerberos support +
small fixes
URL: https://github.com/apache/ambari-logsearch/pull/27
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
index f9ef32d688..a15ac7468a 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
@@ -132,6 +132,8 @@
public static final String HDFS_PORT = "logfeeder.hdfs.port";
public static final String HDFS_FILE_PERMISSIONS =
"logfeeder.hdfs.file.permissions";
public static final String HDFS_KERBEROS = "logfeeder.hdfs.kerberos";
+ public static final String HDFS_KERBEROS_KEYTAB = "logfeeder.hdfs.keytab";
+ public static final String HDFS_KERBEROS_PRINCIPAL =
"logfeeder.hdfs.principal";
public static final String S3_ENDPOINT = "logfeeder.s3.endpoint";
public static final String S3_ENDPOINT_DEFAULT = "https://s3.amazonaws.com";;
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
index f2eb6c741e..b6ab4c7342 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
@@ -19,7 +19,7 @@
package org.apache.ambari.logfeeder.conf;
import org.apache.ambari.logfeeder.common.LogFeederConstants;
-import org.apache.ambari.logfeeder.conf.output.ExternalHdfsOutputConfig;
+import org.apache.ambari.logfeeder.conf.output.HdfsOutputConfig;
import org.apache.ambari.logfeeder.conf.output.RolloverConfig;
import org.apache.ambari.logfeeder.conf.output.S3OutputConfig;
import org.apache.ambari.logfeeder.plugin.common.LogFeederProperties;
@@ -53,7 +53,7 @@
private S3OutputConfig s3OutputConfig;
@Inject
- private ExternalHdfsOutputConfig hdfsOutputConfig;
+ private HdfsOutputConfig hdfsOutputConfig;
private Properties properties;
@@ -258,7 +258,7 @@
defaultValue = "false",
sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
)
- @Value("${" + LogFeederConstants.CLOUD_STORAGE_USE_HDFS_CLIENT + ":false}")
+ @Value("${" + LogFeederConstants.CLOUD_STORAGE_USE_HDFS_CLIENT + ":true}")
private boolean useCloudHdfsClient;
@LogSearchPropertyDescription(
@@ -280,15 +280,6 @@
@Value("${" + LogFeederConstants.CLOUD_STORAGE_BASE_PATH + ":}")
private String cloudBasePath;
- @LogSearchPropertyDescription(
-name = LogFeederConstants.HDFS_USER,
-description = "Overrides HADOOP_USER_NAME variable at runtime",
-examples = {"hdfs"},
-sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
- )
- @Value("${"+ LogFeederConstants.HDFS_USER + ":}")
- private String logfeederHdfsUser;
-
@LogSearchPropertyDescription(
name = LogFeederConstants.CLOUD_STORAGE_USE_FILTERS,
description = "Use filters for inputs (with filters the output format will
be JSON)",
@@ -460,7 +451,7 @@ public void setCloudStorageMode(LogFeederMode
cloudStorageMode) {
this.cloudStorageMode = cloudStorageMode;
}
- public ExternalHdfsOutputConfig getHdfsOutputConfig() {
+ public HdfsOutputConfig getHdfsOutputConfig() {
return hdfsOutputConfig;
}
@@ -480,7 +471,7 @@ public void setRolloverConfig(RolloverConfig
rolloverConfig) {
this.rolloverConfig = rolloverConfig;
}
- public void setHdfsOutputConfig(ExternalHdfsOutputConfig hdfsOutputConfig) {
+ public void setHdfsOutputConfig(HdfsOutputConfig hdfsOutputConfig) {
this.hdfsOutputConfig = hdfsOutputConfig;
}
@@ -512,14 +503,6 @@ public boolean isUseCloudHdfsClient() {
return useCloudHdfsClient;
}
- public String getLogfeederHdfsUser() {
-return logfeederHdfsUser;
- }
-
- public void setLogfeederHdfsUser(String logfeederHdfsUser) {
-this.logfeederHdfsUser = logfeederHdfsUser;
- }
-
public void setUseCloudHdfsClient(boolean useCloudHdfsClient) {
this.useCloudHdfsClient = useCloudHdfsClient;
}
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/ExternalHdfsOutputConfig.jav
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691678#comment-16691678
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #27: AMBARI-24833. HDFS client
kerberos support + small fixes
URL: https://github.com/apache/ambari-logsearch/pull/27#discussion_r234610512
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/HdfsOutputConfig.java
##
@@ -62,7 +71,27 @@
sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
)
@Value("${"+ LogFeederConstants.HDFS_KERBEROS + ":false}")
- private boolean secure;
+ private boolean hdfsKerberos;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.HDFS_KERBEROS_KEYTAB,
+description = "Kerberos keytab location for Log Feeder for communicating
with secure HDFS. ",
+examples = {"/etc/security/keytabs/mykeytab.keytab"},
+defaultValue = "/etc/security/keytabs/logfeeder.service.keytab",
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.HDFS_KERBEROS_KEYTAB +
":/etc/security/keytabs/logfeeder.service.keytab}")
+ private String keytab;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.HDFS_KERBEROS_PRINCIPAL,
+description = "Kerberos keytab location for Log Feeder for communicating
with secure HDFS. ",
Review comment:
This is the same as in line 78
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691477#comment-16691477 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #35 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/35/]) AMBARI-24833. Support for cloud logs to using filters + JSON output (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=72b3dd6d39d233db5b02536b3c6a3215971f6cd6]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/CloudStorageInputConfigHandler.java * (edit) ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java * (edit) ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/AbstractInputConfigHandler.java * (edit) ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputLineEnricher.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/DefaultInputConfigHandler.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManagerImpl.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageOutputManager.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691475#comment-16691475 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #27: AMBARI-24833. HDFS client kerberos support + small fixes URL: https://github.com/apache/ambari-logsearch/pull/27#issuecomment-439831367 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/57/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691466#comment-16691466 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #27: AMBARI-24833. HDFS client kerberos support + small fixes URL: https://github.com/apache/ambari-logsearch/pull/27 # What changes were proposed in this pull request? - add kerberos support for hdfs client - remove external hdfs client -> merge it with hdfs client - small fixes ## How was this patch tested? FT: manually - docker/vagrant/gce env This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691460#comment-16691460
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #26: AMBARI-24833. Support for cloud logs to using
filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
b/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
index f6cb519b95..12af637d8c 100644
---
a/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
+++
b/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
@@ -84,12 +84,7 @@ public void monitorInputConfigChanges(final
InputConfigMonitor inputConfigMonito
File[] inputConfigFiles = new
File(configDir).listFiles(inputConfigFileFilter);
if (inputConfigFiles != null) {
for (File inputConfigFile : inputConfigFiles) {
-String inputConfig = new
String(Files.readAllBytes(inputConfigFile.toPath()));
-Matcher m = serviceNamePattern.matcher(inputConfigFile.getName());
-m.find();
-String serviceName = m.group(1);
-JsonElement inputConfigJson =
JsonHelper.mergeGlobalConfigWithInputConfig(parser, inputConfig,
globalConfigNode);
-inputConfigMonitor.loadInputConfigs(serviceName,
InputConfigGson.gson.fromJson(inputConfigJson, InputConfigImpl.class));
+tryLoadingInputConfig(inputConfigMonitor, parser, globalConfigNode,
inputConfigFile);
}
}
final FileSystem fs = FileSystems.getDefault();
@@ -100,6 +95,41 @@ public void monitorInputConfigChanges(final
InputConfigMonitor inputConfigMonito
executorService.submit(updater);
}
+ private void tryLoadingInputConfig(InputConfigMonitor inputConfigMonitor,
JsonParser parser, JsonArray globalConfigNode, File inputConfigFile) throws
Exception {
+// note: that will try to solve a intermittent issue when the input config
json is a null string (during file generation), that process will re-try to
process the files a few times
+int tries = 0;
+while(true) {
+ tries++;
+ Matcher m = serviceNamePattern.matcher(inputConfigFile.getName());
+ m.find();
+ String inputConfig = new
String(Files.readAllBytes(inputConfigFile.toPath()));
+ String serviceName = m.group(1);
+ JsonElement inputConfigJson = null;
+ logger.info("Trying to load '{}' service input config from input file
'{}'", serviceName, inputConfigFile.getAbsolutePath());
+ try {
+inputConfigJson = JsonHelper.mergeGlobalConfigWithInputConfig(parser,
inputConfig, globalConfigNode);
+ } catch (Exception e) {
+final String errorMessage;
+if (tries < 3) {
+ errorMessage = String.format("Cannot parse input config: %s, will
retry in a few seconds again (tries: %s)", inputConfig, String.valueOf(tries));
+ logger.error(errorMessage, e);
+ try {
+Thread.sleep(2000);
+ } catch (Exception ex) {
+// skip
+ }
+ continue;
+} else {
+ errorMessage = String.format("Cannot parse input config: %s, after
%s tries. Will skip to processing it", inputConfig, String.valueOf(tries));
+ logger.error(errorMessage, e);
+ break;
+}
+ }
+ inputConfigMonitor.loadInputConfigs(serviceName,
InputConfigGson.gson.fromJson(inputConfigJson, InputConfigImpl.class));
+ break;
+}
+ }
+
@Override
public void close() throws IOException {
}
diff --git
a/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java
b/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java
index fd08e07fed..0975c395fa 100644
---
a/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java
+++
b/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java
@@ -48,6 +48,7 @@
public LogLevelFilterManagerZK(Map properties) throws
Exception {
this.client = LogSearchConfigZKHelper.createZKClient(properties);
+this.client.start();
this.serverCache = new TreeCache(client, "/");
this.aclList = LogSearchConfigZKHelper.getA
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691454#comment-16691454
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #26: AMBARI-24833. Support for
cloud logs to using filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26#discussion_r234544396
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputLineEnricher.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output;
+
+import com.google.common.hash.Hashing;
+import org.apache.ambari.logfeeder.plugin.common.MetricData;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.ambari.logfeeder.plugin.input.InputMarker;
+import org.apache.ambari.logfeeder.util.LogFeederUtil;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Utility class for fill output with other fields
+ */
+public class OutputLineEnricher {
+
+ private static final Logger logger =
LogManager.getLogger(OutputLineEnricher.class);
+
+ private static final int MAX_OUTPUT_SIZE = 32765; // 32766-1
+
+ public void enrichFields(final Map jsonObj, final
InputMarker inputMarker, final MetricData messageTruncateMetric) {
+Input input = inputMarker.getInput();
+// Update the block with the context fields
+for (Map.Entry entry :
input.getInputDescriptor().getAddFields().entrySet()) {
+ if (jsonObj.get(entry.getKey()) == null ||
entry.getKey().equals("cluster") && "null".equals(jsonObj.get(entry.getKey(
{
Review comment:
i will solve that in a next commit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691455#comment-16691455
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #26: AMBARI-24833. Support for
cloud logs to using filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26#discussion_r23455
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/AbstractInputConfigHandler.java
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.manager.operations.impl;
+
+import org.apache.ambari.logfeeder.manager.InputConfigHolder;
+import org.apache.ambari.logfeeder.manager.operations.InputConfigHandler;
+import org.apache.ambari.logfeeder.plugin.common.AliasUtil;
+import org.apache.ambari.logfeeder.plugin.filter.Filter;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.FilterDescriptor;
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Holds common operations for input config handlers
+ */
+public abstract class AbstractInputConfigHandler implements InputConfigHandler
{
+
+ private static final Logger logger =
LogManager.getLogger(AbstractInputConfigHandler.class);
+
+ protected void loadFilters(String serviceName, InputConfigHolder
inputConfigHolder) {
+sortFilters(inputConfigHolder);
+
+List toRemoveInputList = new ArrayList<>();
+for (Input input :
inputConfigHolder.getInputManager().getInputList(serviceName)) {
+ for (FilterDescriptor filterDescriptor :
inputConfigHolder.getFilterConfigList()) {
+if (filterDescriptor == null) {
+ logger.warn("Filter descriptor is smpty. Skipping...");
Review comment:
i will solve that in a next commit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691379#comment-16691379
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #26: AMBARI-24833. Support for
cloud logs to using filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26#discussion_r234520580
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/AbstractInputConfigHandler.java
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.manager.operations.impl;
+
+import org.apache.ambari.logfeeder.manager.InputConfigHolder;
+import org.apache.ambari.logfeeder.manager.operations.InputConfigHandler;
+import org.apache.ambari.logfeeder.plugin.common.AliasUtil;
+import org.apache.ambari.logfeeder.plugin.filter.Filter;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.FilterDescriptor;
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Holds common operations for input config handlers
+ */
+public abstract class AbstractInputConfigHandler implements InputConfigHandler
{
+
+ private static final Logger logger =
LogManager.getLogger(AbstractInputConfigHandler.class);
+
+ protected void loadFilters(String serviceName, InputConfigHolder
inputConfigHolder) {
+sortFilters(inputConfigHolder);
+
+List toRemoveInputList = new ArrayList<>();
+for (Input input :
inputConfigHolder.getInputManager().getInputList(serviceName)) {
+ for (FilterDescriptor filterDescriptor :
inputConfigHolder.getFilterConfigList()) {
+if (filterDescriptor == null) {
+ logger.warn("Filter descriptor is smpty. Skipping...");
Review comment:
typo: smpty -> empty
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691380#comment-16691380
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #26: AMBARI-24833. Support for
cloud logs to using filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26#discussion_r234521392
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputLineEnricher.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output;
+
+import com.google.common.hash.Hashing;
+import org.apache.ambari.logfeeder.plugin.common.MetricData;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.ambari.logfeeder.plugin.input.InputMarker;
+import org.apache.ambari.logfeeder.util.LogFeederUtil;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Utility class for fill output with other fields
+ */
+public class OutputLineEnricher {
+
+ private static final Logger logger =
LogManager.getLogger(OutputLineEnricher.class);
+
+ private static final int MAX_OUTPUT_SIZE = 32765; // 32766-1
+
+ public void enrichFields(final Map jsonObj, final
InputMarker inputMarker, final MetricData messageTruncateMetric) {
+Input input = inputMarker.getInput();
+// Update the block with the context fields
+for (Map.Entry entry :
input.getInputDescriptor().getAddFields().entrySet()) {
+ if (jsonObj.get(entry.getKey()) == null ||
entry.getKey().equals("cluster") && "null".equals(jsonObj.get(entry.getKey(
{
Review comment:
```"cluster".equals(entry.getKey())```
would be safer
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690680#comment-16690680 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #26: AMBARI-24833. Support for cloud logs to using filters + JSON output URL: https://github.com/apache/ambari-logsearch/pull/26#issuecomment-439642020 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/56/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690678#comment-16690678 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #26: AMBARI-24833. Support for cloud logs to using filters + JSON output URL: https://github.com/apache/ambari-logsearch/pull/26#issuecomment-439641461 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/55/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689697#comment-16689697 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #26: AMBARI-24833. Support for cloud logs to using filters + JSON output URL: https://github.com/apache/ambari-logsearch/pull/26#issuecomment-439465913 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/54/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689628#comment-16689628 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #26: AMBARI-24833. Support for cloud logs to using filters + JSON output URL: https://github.com/apache/ambari-logsearch/pull/26#issuecomment-439447527 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/53/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689622#comment-16689622 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #26: AMBARI-24833. Support for cloud logs to using filters + JSON output URL: https://github.com/apache/ambari-logsearch/pull/26 # What changes were proposed in this pull request? - optionally use filters (+ json output) for logs that will be archived to cloud storage (default: false) - create output enricher class - only extend common operations there (there is not any refactor there) ## How was this patch tested? FT: manually This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689280#comment-16689280 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #34 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/34/]) AMBARI-24833. Extend cloud log rolling options (#25) (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=31c70d37c30488205d91f666316949d9a7242dd8]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/UploadClientFactory.java * (edit) ambari-logsearch-logfeeder/pom.xml * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/CloudStorageDestination.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/ExternalHdfsOutputConfig.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/ExternalHDFSUploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689276#comment-16689276
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #25: AMBARI-24833. Extend cloud log rolling options
URL: https://github.com/apache/ambari-logsearch/pull/25
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/ambari-logsearch-logfeeder/pom.xml
b/ambari-logsearch-logfeeder/pom.xml
index e71b3cc555..003b23357f 100644
--- a/ambari-logsearch-logfeeder/pom.xml
+++ b/ambari-logsearch-logfeeder/pom.xml
@@ -150,6 +150,12 @@
org.apache.hadoop
hadoop-aws
${hadoop.version}
+
+
+ com.amazonaws
+ aws-java-sdk-bundle
+
+
com.amazonaws
@@ -161,6 +167,11 @@
aws-java-sdk-s3
${aws-sdk.version}
+
+ com.amazonaws
+ aws-java-sdk-dynamodb
+ ${aws-sdk.version}
+
org.apache.hadoop
hadoop-azure
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
index 11d351f23d..b5fffa829b 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
@@ -34,10 +34,6 @@
// Event History Constants History
public static final String VALUES = "jsons";
public static final String ROW_TYPE = "rowtype";
-
- // S3 Constants
- public static final String S3_PATH_START_WITH = "s3://";
- public static final String S3_PATH_SEPARATOR = "/";
public static final String IN_MEMORY_TIMESTAMP = "in_memory_timestamp";
@@ -122,14 +118,17 @@
public static final String CLOUD_ROLLOVER_ARCHIVE_LOCATION =
"logfeeder.cloud.rollover.archive.base.dir";
public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_MIN =
"logfeeder.cloud.rollover.threshold.min";
public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE =
"logfeeder.cloud.rollover.threshold.size";
+ public static final String CLOUD_ROLLOVER_MAX_BACKUP_FILES =
"logfeeder.cloud.rollover.max.files";
+ public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE_UNIT =
"logfeeder.cloud.rollover.threshold.size.unit";
public static final String CLOUD_ROLLOVER_USE_GZIP =
"logfeeder.cloud.rollover.use.gzip";
public static final String CLOUD_ROLLOVER_IMMEDIATE_FLUSH =
"logfeeder.cloud.rollover.immediate.flush";
public static final String CLOUD_ROLLOVER_ON_SHUTDOWN =
"logfeeder.cloud.rollover.on.shutdown";
public static final String CLOUD_ROLLOVER_ON_STARTUP =
"logfeeder.cloud.rollover.on.startup";
+ public static final String HDFS_USER = "logfeeder.hdfs.user";
+
public static final String HDFS_HOST = "logfeeder.hdfs.host";
public static final String HDFS_PORT = "logfeeder.hdfs.port";
- public static final String HDFS_USER = "logfeeder.hdfs.user";
public static final String HDFS_FILE_PERMISSIONS =
"logfeeder.hdfs.file.permissions";
public static final String HDFS_KERBEROS = "logfeeder.hdfs.kerberos";
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/CloudStorageDestination.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/CloudStorageDestination.java
index 1a7eafa509..0246b98306 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/CloudStorageDestination.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/CloudStorageDestination.java
@@ -19,7 +19,7 @@
package org.apache.ambari.logfeeder.conf;
public enum CloudStorageDestination {
- HDFS("hdfs"), S3("s3"), GCS("gcs"), ADLS("adls"), NONE("none");
+ HDFS("hdfs"), S3("s3"), GCS("gcs"), ADLS("adls"), WASB("wasb"),
DEFAULT_FS("default-fs"), NONE("none");
private String text;
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
index d32e1df9db..83f10e497c 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
@@ -224,7 +224,8 @@
@LogSearchPropertyDescription(
name = LogFeederConstants.CLOUD_STORAGE_DESTINATION,
description = "Type of storage that is the destination for cloud output
logs.",
-examples =
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689274#comment-16689274 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #25: AMBARI-24833. Extend cloud log rolling options URL: https://github.com/apache/ambari-logsearch/pull/25#issuecomment-439354665 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/52/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689253#comment-16689253
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #25: AMBARI-24833. Extend cloud
log rolling options
URL: https://github.com/apache/ambari-logsearch/pull/25#discussion_r234152011
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java
##
@@ -49,12 +49,32 @@
@LogSearchPropertyDescription(
name = LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE,
description = "Rollover cloud log files after the log file size reach this
limit",
-examples = {"1024KB"},
-defaultValue = "80MB",
+examples = {"1024"},
+defaultValue = "80",
sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
)
- @Value("${"+ LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE +
":80MB}")
- private String rolloverSize;
+ @Value("${"+ LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE + ":80}")
+ private Integer rolloverSize;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.CLOUD_ROLLOVER_MAX_BACKUP_FILES,
+description = "The number of max backup log files for rolled over logs.",
+examples = {"50"},
+defaultValue = "10",
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.CLOUD_ROLLOVER_MAX_BACKUP_FILES + ":10}")
+ private Integer rolloverMaxBackupFiles;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE_FORMAT,
+description = "Rollover cloud log file size format (e.g: KB, MB etc.)",
Review comment:
Unit or format?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689254#comment-16689254
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #25: AMBARI-24833. Extend cloud
log rolling options
URL: https://github.com/apache/ambari-logsearch/pull/25#discussion_r234151152
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
##
@@ -224,7 +224,8 @@
@LogSearchPropertyDescription(
name = LogFeederConstants.CLOUD_STORAGE_DESTINATION,
description = "Type of storage that is the destination for cloud output
logs.",
-examples = {"hdfs", "s3", "gcs", "adls", "none"},
+examples = {"hdfs", "s3", "gcs", "adls", "wasb", "none"},
Review comment:
Can `default-fs` be set here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689236#comment-16689236 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #25: AMBARI-24833. Extend cloud log rolling options URL: https://github.com/apache/ambari-logsearch/pull/25#issuecomment-439342879 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/51/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688697#comment-16688697 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #25: AMBARI-24833. Extend cloud log rolling options URL: https://github.com/apache/ambari-logsearch/pull/25#issuecomment-439202536 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/50/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688491#comment-16688491
]
ASF GitHub Bot commented on AMBARI-24833:
-
swagle commented on a change in pull request #25: AMBARI-24833. Extend cloud
log rolling options
URL: https://github.com/apache/ambari-logsearch/pull/25#discussion_r233960818
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/CloudStorageDestination.java
##
@@ -19,7 +19,7 @@
package org.apache.ambari.logfeeder.conf;
public enum CloudStorageDestination {
- HDFS("hdfs"), S3("s3"), GCS("gcs"), ADLS("adls"), NONE("none");
+ HDFS("hdfs"), S3("s3"), GCS("gcs"), ADLS("adls"), WASB("wasb"), NONE("none");
Review comment:
Makes sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688488#comment-16688488
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #25: AMBARI-24833. Extend cloud
log rolling options
URL: https://github.com/apache/ambari-logsearch/pull/25#discussion_r233959595
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/CloudStorageDestination.java
##
@@ -19,7 +19,7 @@
package org.apache.ambari.logfeeder.conf;
public enum CloudStorageDestination {
- HDFS("hdfs"), S3("s3"), GCS("gcs"), ADLS("adls"), NONE("none");
+ HDFS("hdfs"), S3("s3"), GCS("gcs"), ADLS("adls"), WASB("wasb"), NONE("none");
Review comment:
it is something for logfeeder level, but as we are not supporting core
gcs/wasb yet it won't be needed anyway, default-fs is used, if hdfs client will
figure out the right filesystem
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688465#comment-16688465
]
ASF GitHub Bot commented on AMBARI-24833:
-
swagle commented on a change in pull request #25: AMBARI-24833. Extend cloud
log rolling options
URL: https://github.com/apache/ambari-logsearch/pull/25#discussion_r233954847
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/CloudStorageDestination.java
##
@@ -19,7 +19,7 @@
package org.apache.ambari.logfeeder.conf;
public enum CloudStorageDestination {
- HDFS("hdfs"), S3("s3"), GCS("gcs"), ADLS("adls"), NONE("none");
+ HDFS("hdfs"), S3("s3"), GCS("gcs"), ADLS("adls"), WASB("wasb"), NONE("none");
Review comment:
Do we need to add secure mode enums? (adlss, wasbs) are the ones that I know
exist.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688436#comment-16688436 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #25: AMBARI-24833. Extend cloud log rolling options URL: https://github.com/apache/ambari-logsearch/pull/25#issuecomment-439130819 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/49/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688330#comment-16688330 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #25: AMBARI-24833. Extend cloud log rolling options URL: https://github.com/apache/ambari-logsearch/pull/25#issuecomment-439106294 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/48/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688317#comment-16688317 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #25: AMBARI-24833. Extend cloud log rolling options URL: https://github.com/apache/ambari-logsearch/pull/25 # What changes were proposed in this pull request? Extend rolling options: - add size format - add default rolling strategy ## How was this patch tested? Waiting for UTs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685012#comment-16685012 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #28 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/28/]) AMBARI-24833. Simplify HDFS client usage + use core-site.xml (#21) (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=db5fb3db496beec489a55704eaf9b58d9d683bac]) * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/AbstractCloudClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/UploadClientFactory.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/UploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/OutputConfig.java * (add) ambari-logsearch-logfeeder/src/main/resources/core-site.xml * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/CloudStorageOutputConfig.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/HdfsOutputConfig.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/S3OutputConfig.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/BucketConfig.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/ExternalHDFSUploadClient.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/ExternalHdfsOutputConfig.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/S3UploadClient.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/AbstractS3CloudClient.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684998#comment-16684998
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #21: AMBARI-24833. Simplify HDFS client usage +
use core-site.xml
URL: https://github.com/apache/ambari-logsearch/pull/21
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
index f528c45a02..11d351f23d 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
@@ -113,9 +113,13 @@
public static final String CLOUD_STORAGE_DESTINATION =
"logfeeder.cloud.storage.destination";
public static final String CLOUD_STORAGE_UPLOAD_ON_SHUTDOWN =
"logfeeder.cloud.storage.upload.on.shutdown";
public static final String CLOUD_STORAGE_UPLOADER_INTERVAL_SECONDS =
"logfeeder.cloud.storage.uploader.interval.seconds";
+ public static final String CLOUD_STORAGE_BUCKET =
"logfeeder.cloud.storage.bucket";
public static final String CLOUD_STORAGE_BUCKET_BOOTSTRAP =
"logfeeder.cloud.storage.bucket.bootstrap";
public static final String CLOUD_STORAGE_USE_HDFS_CLIENT =
"logfeeder.cloud.storage.use.hdfs.client";
+ public static final String CLOUD_STORAGE_CUSTOM_FS =
"logfeeder.cloud.storage.custom.fs";
+ public static final String CLOUD_STORAGE_BASE_PATH =
"logfeeder.cloud.storage.base.path";
+ public static final String CLOUD_ROLLOVER_ARCHIVE_LOCATION =
"logfeeder.cloud.rollover.archive.base.dir";
public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_MIN =
"logfeeder.cloud.rollover.threshold.min";
public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE =
"logfeeder.cloud.rollover.threshold.size";
public static final String CLOUD_ROLLOVER_USE_GZIP =
"logfeeder.cloud.rollover.use.gzip";
@@ -126,14 +130,12 @@
public static final String HDFS_HOST = "logfeeder.hdfs.host";
public static final String HDFS_PORT = "logfeeder.hdfs.port";
public static final String HDFS_USER = "logfeeder.hdfs.user";
- public static final String HDFS_OUTPUT_BASE_DIR =
"logfeeder.hdfs.output.base.dir";
public static final String HDFS_FILE_PERMISSIONS =
"logfeeder.hdfs.file.permissions";
public static final String HDFS_KERBEROS = "logfeeder.hdfs.kerberos";
public static final String S3_ENDPOINT = "logfeeder.s3.endpoint";
public static final String S3_ENDPOINT_DEFAULT = "https://s3.amazonaws.com";;
public static final String S3_REGION = "logfeeder.s3.region";
- public static final String S3_BUCKET = "logfeeder.s3.bucket";
public static final String S3_OBJECT_ACL = "logfeeder.s3.object.acl";
public static final String S3_PATH_STYLE_ACCESS =
"logfeeder.s3.path.style.access";
public static final String S3_MULTIOBJECT_DELETE_ENABLE =
"logfeeder.s3.multiobject.delete.enable";
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
index e89f7f4ca0..d32e1df9db 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
@@ -19,7 +19,7 @@
package org.apache.ambari.logfeeder.conf;
import org.apache.ambari.logfeeder.common.LogFeederConstants;
-import org.apache.ambari.logfeeder.conf.output.HdfsOutputConfig;
+import org.apache.ambari.logfeeder.conf.output.ExternalHdfsOutputConfig;
import org.apache.ambari.logfeeder.conf.output.RolloverConfig;
import org.apache.ambari.logfeeder.conf.output.S3OutputConfig;
import org.apache.ambari.logfeeder.plugin.common.LogFeederProperties;
@@ -47,13 +47,13 @@
private Environment env;
@Inject
- private HdfsOutputConfig hdfsOutputConfig;
+ private RolloverConfig rolloverConfig;
@Inject
private S3OutputConfig s3OutputConfig;
@Inject
- private RolloverConfig rolloverConfig;
+ private ExternalHdfsOutputConfig hdfsOutputConfig;
private Properties properties;
@@ -238,7 +238,7 @@
sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
)
@Value("${" + LogFeederConstants.CLOUD_STORAGE_UPLOAD_ON_SHUTDOWN +
":false}")
- public boolean cloudStorageUploadOnShutdown;
+ private boolean cloudStorageUploadOnShutdown;
@LogSearchPropertyD
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684556#comment-16684556 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #21: AMBARI-24833. Simplify HDFS client usage + use core-site.xml URL: https://github.com/apache/ambari-logsearch/pull/21#issuecomment-438083055 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/43/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684548#comment-16684548 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #21: AMBARI-24833. Simplify HDFS client usage + use core-site.xml URL: https://github.com/apache/ambari-logsearch/pull/21 # What changes were proposed in this pull request? - refactor hdfs / s3 clients - use hdfs clients with core-site.xml (and other files) from loading them to classpath (it will just check core-site.xml existence) - as gcs connector and wasb/adls are on the classpath those should work as well with the right configs - remove some unnecessary abstractions after refactor - be able to customize active/archive base dir location + override defaultFS from logfeeder config ## How was this patch tested? (Please explain how this patch was tested. Ex: unit tests, manual tests) (If this patch involves UI changes, please attach a screen-shot; otherwise, remove this) Please review [Ambari Contributing Guide](https://cwiki.apache.org/confluence/display/AMBARI/How+to+Contribute) before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683822#comment-16683822 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #27 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/27/]) AMBARI-24833. Re-implement S3/HDFS outputs as global cloud outputs (#19) (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=2bd96e130795d1a1b7961ad5cc695487df05af07]) * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java * (edit) ambari-logsearch-logfeeder-plugin-api/src/main/java/org/apache/ambari/logfeeder/plugin/manager/OutputManager.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/S3Uploader.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/CloudStorageDestination.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/SecretKeyPair.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageOutputManager.java * (edit) ambari-logsearch-logfeeder/src/main/resources/alias_config.json * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/S3OutputConfiguration.java * (edit) ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties * (edit) ambari-logsearch-logfeeder-plugin-api/src/main/java/org/apache/ambari/logfeeder/plugin/output/Output.java * (edit) docker/test-config/logfeeder/logfeeder.properties * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/HDFSOutput.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/S3UploadClient.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/LogSpoolerException.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/UploadClientFactory.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputS3File.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageFactory.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/RolloverHandler.java * (delete) ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/S3LogPathResolverTest.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/LogSpooler.java * (edit) docker/cloud-docker-compose.yml * (edit) ambari-logsearch-server/pom.xml * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/AbstractCloudClient.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/credential/PlainTextSecretStore.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/OutputConfig.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/UploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/BucketConfig.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/spool/RolloverCondition.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/S3Util.java * (edit) ambari-logsearch-logfeeder-plugin-api/src/main/java/org/apache/ambari/logfeeder/plugin/input/Input.java * (delete) ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/spool/LogSpoolerTest.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/S3OutputConfig.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/HdfsOutputConfig.java * (delete) ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/S3UploaderTest.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/CloudStorageOutputConfig.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/S3LogPathResolver.java * (edit) ambari-l
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683806#comment-16683806
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232661059
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageOutput.java
##
@@ -19,12 +19,139 @@
package org.apache.ambari.logfeeder.output.cloud;
import org.apache.ambari.logfeeder.conf.LogFeederProps;
+import org.apache.ambari.logfeeder.conf.output.RolloverConfig;
+import org.apache.ambari.logfeeder.output.cloud.upload.UploadClient;
+import org.apache.ambari.logfeeder.output.cloud.upload.UploadClientFactory;
+import org.apache.ambari.logfeeder.plugin.input.Input;
import org.apache.ambari.logfeeder.plugin.input.InputMarker;
import org.apache.ambari.logfeeder.plugin.output.Output;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.core.Appender;
+import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.appender.RollingFileAppender;
+import org.apache.logging.log4j.core.config.Configuration;
+
+import java.io.File;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
/**
- * Class to handle common operations for cloud storage outputs
- * TODO !!!
+ * Output class for cloud outputs.
+ * Holds loggers - those will ship logs into specific folders, those files can
be rolled out to an archive folder,
+ * from there an upload client will be able to ship the log archives to a
cloud storage
*/
-public abstract class CloudStorageOutput extends Output {
+public class CloudStorageOutput extends Output {
+
+ private static final Logger logger =
LogManager.getLogger(CloudStorageOutput.class);
+
+ private final Map cloudOutputLoggers = new
ConcurrentHashMap<>();
+ private final UploadClient uploadClient;
+ private final LogFeederProps logFeederProps;
+ private final LoggerContext loggerContext;
+ private final CloudStorageUploader uploader;
+ private final RolloverConfig rolloverConfig;
+
+ public CloudStorageOutput(LogFeederProps logFeederProps) {
+this.uploadClient = UploadClientFactory.createUploadClient(logFeederProps);
+this.logFeederProps = logFeederProps;
+this.rolloverConfig = logFeederProps.getRolloverConfig();
+loggerContext = (LoggerContext) LogManager.getContext(false);
+uploader = new CloudStorageUploader(String.format("%s-uploader",
logFeederProps.getCloudStorageDestination().getText()), uploadClient,
logFeederProps);
+uploader.setDaemon(true);
+ }
+
+ @Override
+ public void init(LogFeederProps logFeederProperties) throws Exception {
+logger.info("Initialize cloud output.");
+uploadClient.init(logFeederProperties);
+uploader.start();
+ }
+
+ @Override
+ public String getOutputType() {
+return "cloud";
+ }
+
+ @Override
+ public void copyFile(File inputFile, InputMarker inputMarker) throws
Exception {
+throw new UnsupportedOperationException("Copy file is not supported yet");
+ }
+
+ @Override
+ public void write(String jsonStr, InputMarker inputMarker) throws Exception {
+String uniqueThreadName = inputMarker.getInput().getThread().getName();
+Logger cloudLogger = null;
+if (cloudOutputLoggers.containsKey(uniqueThreadName)) {
+ cloudLogger = cloudOutputLoggers.get(uniqueThreadName);
+} else {
+ logger.info("New cloud input source found. Register: {}",
uniqueThreadName);
+ cloudLogger =
CloudStorageLoggerFactory.createLogger(inputMarker.getInput(), loggerContext,
logFeederProps);
+ cloudOutputLoggers.put(uniqueThreadName, cloudLogger);
+}
+cloudLogger.info(jsonStr);
+inputMarker.getInput().checkIn(inputMarker);
+ }
+
+ @Override
+ public Long getPendingCount() {
+return 0L;
+ }
+
+ @Override
+ public String getWriteBytesMetricName() {
+return "write:cloud";
+ }
+
+ @Override
+ public String getShortDescription() {
+return "cloud";
+ }
+
+ @Override
+ public String getStatMetricName() {
+return null;
Review comment:
null is handled
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Is
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683811#comment-16683811 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #19: AMBARI-24833. Re-implement S3/HDFS outputs as global cloud outputs URL: https://github.com/apache/ambari-logsearch/pull/19#issuecomment-437886646 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/37/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683382#comment-16683382
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232542736
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
##
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output.cloud;
+
+import org.apache.ambari.logfeeder.common.LogFeederConstants;
+import org.apache.ambari.logfeeder.conf.LogFeederProps;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.appender.RollingFileAppender;
+import
org.apache.logging.log4j.core.appender.rolling.CompositeTriggeringPolicy;
+import
org.apache.logging.log4j.core.appender.rolling.OnStartupTriggeringPolicy;
+import
org.apache.logging.log4j.core.appender.rolling.SizeBasedTriggeringPolicy;
+import org.apache.logging.log4j.core.config.AppenderRef;
+import org.apache.logging.log4j.core.config.Configuration;
+import org.apache.logging.log4j.core.config.LoggerConfig;
+import org.apache.logging.log4j.core.layout.PatternLayout;
+
+import java.io.File;
+
+/**
+ * Create a custom logger that will be used to ship inputs into specific files
+ * and rollover those files to an archive folder that will be monitored by a
+ * thread which will upload archived files to a cloud storage
+ */
+public class CloudStorageLoggerFactory {
+
+ private static final String ACTIVE_FOLDER = "active";
+ private static final String ARCHIVED_FOLDER = "archived";
+ private static final String DATE_PATTERN_SUFFIX_GZ =
"-%d{-MM-dd-hh-mm-ss-SSS}.log.gz";
+ private static final String DATE_PATTERN_SUFFIX =
"-%d{-MM-dd-hh-mm-ss-SSS}.log";
+
+ public static Logger createLogger(Input input, LoggerContext loggerContext,
LogFeederProps logFeederProps) {
+String type = input.getLogType().replace(LogFeederConstants.CLOUD_PREFIX,
"");
+String uniqueThreadName = input.getThread().getName();
+Configuration config = loggerContext.getConfiguration();
+String destination = logFeederProps.getCloudStorageDestination().getText();
+String activeLogDir = String.join(File.separator,
logFeederProps.getTmpDir(), destination, ACTIVE_FOLDER, type);
Review comment:
Paths.get() can be used too for joining path fragments:
```Paths.get(logFeederProps.getTmpDir(), destination, ACTIVE_FOLDER, type)```
but joining strings is also good
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683383#comment-16683383
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232543287
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageOutput.java
##
@@ -19,12 +19,139 @@
package org.apache.ambari.logfeeder.output.cloud;
import org.apache.ambari.logfeeder.conf.LogFeederProps;
+import org.apache.ambari.logfeeder.conf.output.RolloverConfig;
+import org.apache.ambari.logfeeder.output.cloud.upload.UploadClient;
+import org.apache.ambari.logfeeder.output.cloud.upload.UploadClientFactory;
+import org.apache.ambari.logfeeder.plugin.input.Input;
import org.apache.ambari.logfeeder.plugin.input.InputMarker;
import org.apache.ambari.logfeeder.plugin.output.Output;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.core.Appender;
+import org.apache.logging.log4j.core.LoggerContext;
+import org.apache.logging.log4j.core.appender.RollingFileAppender;
+import org.apache.logging.log4j.core.config.Configuration;
+
+import java.io.File;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
/**
- * Class to handle common operations for cloud storage outputs
- * TODO !!!
+ * Output class for cloud outputs.
+ * Holds loggers - those will ship logs into specific folders, those files can
be rolled out to an archive folder,
+ * from there an upload client will be able to ship the log archives to a
cloud storage
*/
-public abstract class CloudStorageOutput extends Output {
+public class CloudStorageOutput extends Output {
+
+ private static final Logger logger =
LogManager.getLogger(CloudStorageOutput.class);
+
+ private final Map cloudOutputLoggers = new
ConcurrentHashMap<>();
+ private final UploadClient uploadClient;
+ private final LogFeederProps logFeederProps;
+ private final LoggerContext loggerContext;
+ private final CloudStorageUploader uploader;
+ private final RolloverConfig rolloverConfig;
+
+ public CloudStorageOutput(LogFeederProps logFeederProps) {
+this.uploadClient = UploadClientFactory.createUploadClient(logFeederProps);
+this.logFeederProps = logFeederProps;
+this.rolloverConfig = logFeederProps.getRolloverConfig();
+loggerContext = (LoggerContext) LogManager.getContext(false);
+uploader = new CloudStorageUploader(String.format("%s-uploader",
logFeederProps.getCloudStorageDestination().getText()), uploadClient,
logFeederProps);
+uploader.setDaemon(true);
+ }
+
+ @Override
+ public void init(LogFeederProps logFeederProperties) throws Exception {
+logger.info("Initialize cloud output.");
+uploadClient.init(logFeederProperties);
+uploader.start();
+ }
+
+ @Override
+ public String getOutputType() {
+return "cloud";
+ }
+
+ @Override
+ public void copyFile(File inputFile, InputMarker inputMarker) throws
Exception {
+throw new UnsupportedOperationException("Copy file is not supported yet");
+ }
+
+ @Override
+ public void write(String jsonStr, InputMarker inputMarker) throws Exception {
+String uniqueThreadName = inputMarker.getInput().getThread().getName();
+Logger cloudLogger = null;
+if (cloudOutputLoggers.containsKey(uniqueThreadName)) {
+ cloudLogger = cloudOutputLoggers.get(uniqueThreadName);
+} else {
+ logger.info("New cloud input source found. Register: {}",
uniqueThreadName);
+ cloudLogger =
CloudStorageLoggerFactory.createLogger(inputMarker.getInput(), loggerContext,
logFeederProps);
+ cloudOutputLoggers.put(uniqueThreadName, cloudLogger);
+}
+cloudLogger.info(jsonStr);
+inputMarker.getInput().checkIn(inputMarker);
+ }
+
+ @Override
+ public Long getPendingCount() {
+return 0L;
+ }
+
+ @Override
+ public String getWriteBytesMetricName() {
+return "write:cloud";
+ }
+
+ @Override
+ public String getShortDescription() {
+return "cloud";
+ }
+
+ @Override
+ public String getStatMetricName() {
+return null;
Review comment:
Should an `UnsupportedOperationException` be thrown?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
>
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683384#comment-16683384
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232563856
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output.cloud.upload;
+
+import org.apache.ambari.logfeeder.conf.LogFeederProps;
+import org.apache.ambari.logfeeder.conf.output.S3OutputConfig;
+import org.apache.ambari.logfeeder.util.LogFeederHDFSUtil;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.IOException;
+
+public class HDFSS3UploadClient extends AbstractCloudClient implements
UploadClient {
+
+ private static final Logger logger =
LogManager.getLogger(HDFSS3UploadClient.class);
+
+ private final S3OutputConfig s3OutputConfig;
+
+ private FileSystem fs;
+
+ public HDFSS3UploadClient(S3OutputConfig s3OutputConfig) {
+this.s3OutputConfig = s3OutputConfig;
+ }
+
+ @Override
+ void createBucketIfNeeded(String bucket) {
+logger.warn("HDFS based S3 client won't bootstrap default bucket ('{}')",
s3OutputConfig.getBucket());
+ }
+
+ @Override
+ public void init(LogFeederProps logFeederProps) {
+SecretKeyPair keyPair = getSecretKeyPair(logFeederProps, s3OutputConfig);
+// TODO: load configuration from file
+Configuration conf =
LogFeederHDFSUtil.buildHdfsConfiguration(s3OutputConfig.getBucket(), "s3a");
+conf.set("fs.s3a.access.key", new String(keyPair.getAccessKey()));
+conf.set("fs.s3a.secret.key", new String(keyPair.getSecretKey()));
+conf.set("fs.s3a.aws.credentials.provider",
SimpleAWSCredentialsProvider.NAME);
+conf.set("fs.s3a.endpoint", s3OutputConfig.getEndpoint());
+conf.set("fs.s3a.path.style.access",
String.valueOf(s3OutputConfig.isPathStyleAccess()));
+conf.set("fs.s3a.multiobjectdelete.enable",
String.valueOf(s3OutputConfig.isMultiobjectDeleteEnable()));
+this.fs = LogFeederHDFSUtil.buildFileSystem(conf);
+ }
+
+ @Override
+ public boolean upload(String source, String target, String basePath) {
+return LogFeederHDFSUtil.copyFromLocal(source, target, this.fs, true,
true, null);
+ }
+
+ @Override
+ public S3OutputConfig getOutputConfig() {
+return this.s3OutputConfig;
+ }
+
+ @Override
+ public void close() throws IOException {
+if (fs != null) {
Review comment:
You can use
```org.apache.commons.io.IOUtils.closeQuietly(fs);```
if logging the IOException is not necessary
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683386#comment-16683386
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232564695
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/S3UploadClient.java
##
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output.cloud.upload;
+
+import com.amazonaws.auth.AWSCredentials;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3ClientBuilder;
+import com.amazonaws.services.s3.model.CannedAccessControlList;
+import org.apache.ambari.logfeeder.conf.LogFeederProps;
+import org.apache.ambari.logfeeder.conf.output.S3OutputConfig;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.tools.ant.util.FileUtils;
+
+import java.io.File;
+
+/**
+ * S3 specific upload client
+ */
+public class S3UploadClient extends AbstractCloudClient implements
UploadClient {
+
+ private static final Logger logger =
LogManager.getLogger(S3UploadClient.class);
+
+ private final S3OutputConfig s3OutputConfig;
+ private final CannedAccessControlList acl;
+ private AmazonS3 s3Client;
+
+ public S3UploadClient(S3OutputConfig s3OutputConfig) {
+this.s3OutputConfig = s3OutputConfig;
+this.acl = s3OutputConfig.calculateAcls(s3OutputConfig.getObjectAcl());
+ }
+
+ @Override
+ public void init(LogFeederProps logFeederProps) {
+SecretKeyPair keyPair = getSecretKeyPair(logFeederProps, s3OutputConfig);
+AWSCredentials awsCredentials = new BasicAWSCredentials(new
String(keyPair.getAccessKey()), new String(keyPair.getSecretKey()));
+AWSCredentialsProvider credentialsProvider = new
AWSStaticCredentialsProvider(awsCredentials);
+AwsClientBuilder.EndpointConfiguration endpointConf = new
AwsClientBuilder.EndpointConfiguration(s3OutputConfig.getEndpoint(),
s3OutputConfig.getRegion());
+s3Client = AmazonS3ClientBuilder.standard()
+ .withCredentials(credentialsProvider)
+ .withEndpointConfiguration(endpointConf)
+ .withPathStyleAccessEnabled(s3OutputConfig.isPathStyleAccess())
+ .build();
+bootstrapBucket(s3OutputConfig.getOutputBasePath(),
s3OutputConfig.getBucketConfig());
+ }
+
+ @Override
+ public boolean upload(String source, String target, String bucket) {
+boolean result = false;
+try {
+ File fileToUpload = new File(source);
+ logger.info("Starting S3 upload {} -> bucket: {}, key: {}", source,
bucket, target);
+ s3Client.putObject(bucket, target, new File(source));
+ s3Client.setObjectAcl(bucket, target, acl);
+ FileUtils.delete(fileToUpload);
+ result = true;
+} catch (Exception e) {
+ logger.error("Error during upload file to s3.", e);
Review comment:
Why does a logical return value used for indicating exceptions instead of
letting the Exception thrown out of this method?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>As
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683385#comment-16683385
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232536877
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/S3OutputConfig.java
##
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.conf.output;
+
+import com.amazonaws.services.s3.model.CannedAccessControlList;
+import org.apache.ambari.logfeeder.common.LogFeederConstants;
+import org.apache.ambari.logsearch.config.api.LogSearchPropertyDescription;
+import org.springframework.beans.factory.annotation.Value;
+import org.springframework.context.annotation.Configuration;
+
+import javax.inject.Inject;
+
+@Configuration
+public class S3OutputConfig implements CloudStorageOutputConfig {
+
+ private final BucketConfig bucketConfig;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.S3_ENDPOINT,
+description = "Amazon S3 endpoint.",
+examples = {"https://s3.amazonaws.com"},
+defaultValue = LogFeederConstants.S3_ENDPOINT_DEFAULT,
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.S3_ENDPOINT + ":" +
LogFeederConstants.S3_ENDPOINT_DEFAULT +"}")
+ private String endpoint;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.S3_SECRET_KEY,
+description = "Amazon S3 secret key.",
+examples = {"MySecretKey"},
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.S3_SECRET_KEY + ":}")
+ private String secretKey;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.S3_ACCESS_KEY,
+description = "Amazon S3 secret access key.",
+examples = {"MySecretAccessKey"},
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.S3_ACCESS_KEY + ":}")
+ private String accessKey;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.S3_SECRET_KEY_FILE,
+description = "Amazon S3 secret key file (that contains only the key).",
+examples = {"/my/path/secret_key"},
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.S3_ACCESS_KEY + ":}")
+ private String secretKeyFileLocation;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.S3_ACCESS_KEY_FILE,
+description = "Amazon S3 secret access key file (that contains only the
key).",
+examples = {"/my/path/access_key"},
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.S3_ACCESS_KEY_FILE + ":}")
+ private String accessKeyFileLocation;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.S3_USE_FILE,
+description = "Enable to get Amazon S3 secret/access keys from files.",
+examples = {"true"},
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.S3_USE_FILE + ":false}")
+ private boolean useFileSecrets;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.S3_USE_HADOOP_CREDENTIAL_PROVIDER,
+description = "Enable to get Amazon S3 secret/access keys from Hadoop
credential store API.",
+examples = {"true"},
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.S3_USE_HADOOP_CREDENTIAL_PROVIDER +
":false}")
+ private boolean useHadoopCredentialStorage;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.S3_HADOOP_CREDENTIAL_SECRET_REF,
+description = "Amazon S3 secret access key reference in Hadoop credential
store..",
+examples = {"logfeeder.s3.secret.key"},
+defaultValue = "logfeeder.s3.secret.key",
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.S3_HADOOP_CREDENTIAL_SECRET_REF +
":log
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683387#comment-16683387
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232562608
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/AbstractCloudClient.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output.cloud.upload;
+
+import org.apache.ambari.logfeeder.conf.LogFeederProps;
+import org.apache.ambari.logfeeder.conf.output.BucketConfig;
+import org.apache.ambari.logfeeder.conf.output.CloudStorageOutputConfig;
+import org.apache.ambari.logfeeder.credential.CompositeSecretStore;
+import org.apache.ambari.logfeeder.credential.EnvSecretStore;
+import org.apache.ambari.logfeeder.credential.FileSecretStore;
+import org.apache.ambari.logfeeder.credential.HadoopCredentialSecretStore;
+import org.apache.ambari.logfeeder.credential.PlainTextSecretStore;
+import org.apache.ambari.logfeeder.credential.PropertySecretStore;
+import org.apache.ambari.logfeeder.credential.SecretStore;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Holds common cloud based client operations
+ */
+abstract class AbstractCloudClient {
+
+ private static final Logger logger =
LogManager.getLogger(AbstractCloudClient.class);
+
+ /**
+ * Create a cloud specific bucket if it does not exists
+ * @param bucket name of the bucket
+ */
+ abstract void createBucketIfNeeded(String bucket);
+
+ /**
+ * Try to bootstrap the default bucket, until it is not successful.
+ * @param bucket name of the bucket
+ * @param bucketConfig bucket config holder
+ */
+ void bootstrapBucket(String bucket, BucketConfig bucketConfig) {
+if (bucketConfig.isCreateBucketOnStartup()) {
+ boolean ready = false;
+ while (!ready) {
+try {
+ createBucketIfNeeded(bucket);
+ ready = true;
+} catch (Exception e) {
+ logger.error("Error during bucket creation - bucket : " + bucket, e);
+}
+if (!ready) {
+ try {
+Thread.sleep(5000);
+ } catch (InterruptedException e) {
+logger.error("Error during thread sleep (bucket bootstrap)", e);
Review comment:
Set the interrupted flag back to true
```Thread.currentThread().interrupt();```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681884#comment-16681884 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #19: AMBARI-24833. Re-implement S3/HDFS outputs as global cloud outputs URL: https://github.com/apache/ambari-logsearch/pull/19#issuecomment-437473332 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/34/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681870#comment-16681870
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232367451
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output.cloud;
+
+import org.apache.ambari.logfeeder.conf.LogFeederProps;
+import org.apache.ambari.logfeeder.output.cloud.upload.UploadClient;
+import org.apache.ambari.logfeeder.util.LogFeederUtil;
+import org.apache.commons.io.FileUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.File;
+import java.util.Collection;
+
+/**
+ * Periodically checks a folder (contains archived logs) and if it finds any
.log or .gz files, it will try to upload them to cloud storage by an upload
client (cloud specific)
+ */
+public class CloudStorageUploader extends Thread {
+
+ private static final Logger logger =
LogManager.getLogger(CloudStorageUploader.class);
+
+ private final UploadClient uploadClient;
+ private final LogFeederProps logFeederProps;
+ private final String clusterName;
+ private final String hostName;
+ private final String uploaderType;
+
+ public CloudStorageUploader(String name, UploadClient uploadClient,
LogFeederProps logFeederProps) {
+super(name);
+this.uploadClient = uploadClient;
+this.logFeederProps = logFeederProps;
+this.uploaderType = logFeederProps.getCloudStorageDestination().getText();
+this.clusterName = logFeederProps.getClusterName();
+this.hostName = LogFeederUtil.hostName;
+ }
+
+ @Override
+ public void run() {
+logger.info("Start '{}' uploader", uploaderType);
+boolean stop = false;
+do {
+ try {
+doUpload();
+Thread.sleep(1000 *
logFeederProps.getCloudStorageUploaderIntervalSeconds());
+ } catch (InterruptedException iex) {
+logger.info("Uploader ({}) thread interrupted", uploaderType);
+stop = true;
+ }
+} while (!stop && !Thread.currentThread().isInterrupted());
+ }
+
+ /**
+ * Finds .log and .gz files and upload them to cloud storage by an uploader
client
+ */
+ void doUpload() {
+try {
+ final String archiveLogDir = String.join(File.separator,
logFeederProps.getTmpDir(), uploaderType, "archived");
+ if (new File(archiveLogDir).exists()) {
+String[] extensions = {"log", "gz"};
+Collection filesToUpload = FileUtils.listFiles(new
File(archiveLogDir), extensions, true);
+if (filesToUpload.isEmpty()) {
+ logger.debug("Not found any files to upload.");
+} else {
+ for (File file : filesToUpload) {
+String basePath =
uploadClient.getOutputConfig().getOutputBasePath();
+String outputPath = String.format("%s/%s/%s/%s", clusterName,
hostName, file.getParentFile().getName(), file.getName())
+ .replaceAll("//", "/");
+logger.info("Upload will start: input: {}, output: {}",
file.getAbsolutePath(), outputPath);
+uploadClient.upload(file.getAbsolutePath(), outputPath, basePath);
Review comment:
that is handled by the uploadClient implementations, that is not extended to
a method as with hdfs client that can be a parameter of the copy command
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://i
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681805#comment-16681805
]
ASF GitHub Bot commented on AMBARI-24833:
-
swagle commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232349451
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output.cloud;
+
+import org.apache.ambari.logfeeder.conf.LogFeederProps;
+import org.apache.ambari.logfeeder.output.cloud.upload.UploadClient;
+import org.apache.ambari.logfeeder.util.LogFeederUtil;
+import org.apache.commons.io.FileUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.File;
+import java.util.Collection;
+
+/**
+ * Periodically checks a folder (contains archived logs) and if it finds any
.log or .gz files, it will try to upload them to cloud storage by an upload
client (cloud specific)
+ */
+public class CloudStorageUploader extends Thread {
+
+ private static final Logger logger =
LogManager.getLogger(CloudStorageUploader.class);
+
+ private final UploadClient uploadClient;
+ private final LogFeederProps logFeederProps;
+ private final String clusterName;
+ private final String hostName;
+ private final String uploaderType;
+
+ public CloudStorageUploader(String name, UploadClient uploadClient,
LogFeederProps logFeederProps) {
+super(name);
+this.uploadClient = uploadClient;
+this.logFeederProps = logFeederProps;
+this.uploaderType = logFeederProps.getCloudStorageDestination().getText();
+this.clusterName = logFeederProps.getClusterName();
+this.hostName = LogFeederUtil.hostName;
+ }
+
+ @Override
+ public void run() {
+logger.info("Start '{}' uploader", uploaderType);
+boolean stop = false;
+do {
+ try {
+doUpload();
+Thread.sleep(1000 *
logFeederProps.getCloudStorageUploaderIntervalSeconds());
+ } catch (InterruptedException iex) {
+logger.info("Uploader ({}) thread interrupted", uploaderType);
+stop = true;
+ }
+} while (!stop && !Thread.currentThread().isInterrupted());
+ }
+
+ /**
+ * Finds .log and .gz files and upload them to cloud storage by an uploader
client
+ */
+ void doUpload() {
+try {
+ final String archiveLogDir = String.join(File.separator,
logFeederProps.getTmpDir(), uploaderType, "archived");
+ if (new File(archiveLogDir).exists()) {
+String[] extensions = {"log", "gz"};
+Collection filesToUpload = FileUtils.listFiles(new
File(archiveLogDir), extensions, true);
+if (filesToUpload.isEmpty()) {
+ logger.debug("Not found any files to upload.");
+} else {
+ for (File file : filesToUpload) {
+String basePath =
uploadClient.getOutputConfig().getOutputBasePath();
+String outputPath = String.format("%s/%s/%s/%s", clusterName,
hostName, file.getParentFile().getName(), file.getName())
+ .replaceAll("//", "/");
+logger.info("Upload will start: input: {}, output: {}",
file.getAbsolutePath(), outputPath);
+uploadClient.upload(file.getAbsolutePath(), outputPath, basePath);
Review comment:
Shouldn't the archives be deleted after upload? Or filter out ones already
uploaded?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Am
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681778#comment-16681778 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #19: AMBARI-24833. Re-implement S3/HDFS outputs as global cloud outputs URL: https://github.com/apache/ambari-logsearch/pull/19#issuecomment-437444713 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/33/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681771#comment-16681771 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #19: AMBARI-24833. Re-implement S3/HDFS outputs as global cloud outputs URL: https://github.com/apache/ambari-logsearch/pull/19#issuecomment-437443037 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/32/ Test FAILed. Test FAILured. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681736#comment-16681736
]
ASF GitHub Bot commented on AMBARI-24833:
-
swagle commented on a change in pull request #19: AMBARI-24833. Re-implement
S3/HDFS outputs as global cloud outputs
URL: https://github.com/apache/ambari-logsearch/pull/19#discussion_r232333295
##
File path: ambari-logsearch-logfeeder/pom.xml
##
@@ -145,9 +145,19 @@
${ambari-metrics.version}
- io.minio
- minio
- 5.0.1
+ org.apache.hadoop
+ hadoop-aws
+ 3.1.1
+
+
+ com.amazonaws
+ aws-java-sdk-core
+ 1.11.445
Review comment:
This should ideally be parameterized to help build scripts?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681730#comment-16681730 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #19: AMBARI-24833. Re-implement S3/HDFS outputs as global cloud outputs URL: https://github.com/apache/ambari-logsearch/pull/19#issuecomment-437432187 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/31/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16681726#comment-16681726 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #19: AMBARI-24833. Re-implement S3/HDFS outputs as global cloud outputs URL: https://github.com/apache/ambari-logsearch/pull/19 # What changes were proposed in this pull request? - ger rid of minio (reason: as it's really abstract solution for s3 backends, the policy handling is done by plain strings, not with objects) - add hdfs client based and plain s3 outputs (as global configs) - add hdfs output - add a lot of new configs for s3 and hdfs - delete s3 input/output, and hdfs output, delete log spooler framework as well Cloud log process implementation: - create specific loggers for every inputs - rollover those logs into a different folder (archived) - new thread which monitors the archive folders, if there is a new file, upload those to cloud storage - we will have different configurations, to make sure logs are uploaded during shutdown (or even startup) TODOs: - load hadoop configs from file - log/archive directories should not be in tmp folder, they should be configurable No UTs yet ## How was this patch tested? Manually with docker env + on local cluster. waiting UTs to pass Please review @g-boros This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671560#comment-16671560 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #22 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/22/]) AMBARI-24833. Create cloud input/output skeleton. (#17) (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=9bce50d0de0311d106a8c5acaa0f1c68e7097cfc]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/ApplicationConfig.java * (edit) ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/S3UploaderTest.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/credential/EnvSecretStore.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageOutputManager.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/CloudStorageInputConfigHandler.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageFactory.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/HDFSOutput.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java * (edit) docker/test-config/logfeeder/logfeeder.properties * (edit) ambari-logsearch-logfeeder-plugin-api/src/main/java/org/apache/ambari/logfeeder/plugin/input/Input.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederSecurityConfig.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/credential/CompositeSecretStore.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputManagerImpl.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/ConfigHandler.java * (edit) ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/LogSearch.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederMode.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/metrics/StatsLogger.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/InputConfigHandler.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/InputConfigHolder.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/condition/NonCloudStorageCondition.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/BlockMerger.java * (edit) ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/spool/LogSpoolerTest.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/InputConfigManager.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterDummy.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/credential/SecretStore.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/credential/HadoopCredentialSecretStore.java * (edit) ambari-logsearch-logfeeder/src/test/java/org/apache/ambari/logfeeder/output/OutputS3FileTest.java * (edit) ambari-logsearch-logfeeder/pom.xml * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageOutput.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/condition/CloudStorageCondition.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/credential/FileSecretStore.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/DefaultInputConfigHandler.java * (edit) ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/credential/PropertySecretStore.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputSocket.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputConfigUploader.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManagerImpl.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputFile.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671556#comment-16671556
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #17: AMBARI-24833. Create cloud input/output
skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder-plugin-api/src/main/java/org/apache/ambari/logfeeder/plugin/input/Input.java
b/ambari-logsearch-logfeeder-plugin-api/src/main/java/org/apache/ambari/logfeeder/plugin/input/Input.java
index 6228637afc..9ee4533485 100644
---
a/ambari-logsearch-logfeeder-plugin-api/src/main/java/org/apache/ambari/logfeeder/plugin/input/Input.java
+++
b/ambari-logsearch-logfeeder-plugin-api/src/main/java/org/apache/ambari/logfeeder/plugin/input/Input.java
@@ -71,6 +71,7 @@
private LRUCache cache;
private String cacheKeyField;
private boolean initDefaultFields;
+ private boolean cloudInput = false;
private MetricData readBytesMetric = new
MetricData(getReadBytesMetricName(), false);
/**
@@ -400,4 +401,17 @@ public boolean isInitDefaultFields() {
public void setInitDefaultFields(boolean initDefaultFields) {
this.initDefaultFields = initDefaultFields;
}
+
+ public boolean isCloudInput() {
+return cloudInput;
+ }
+
+ public void setCloudInput(boolean cloudInput) {
+this.cloudInput = cloudInput;
+ }
+
+ public String getCloudModeSuffix() {
+String mode = isCloudInput() ? "cloud": "default";
+return "mode=" + mode;
+ }
}
diff --git a/ambari-logsearch-logfeeder/pom.xml
b/ambari-logsearch-logfeeder/pom.xml
index 94af44f3f8..71cf853c18 100644
--- a/ambari-logsearch-logfeeder/pom.xml
+++ b/ambari-logsearch-logfeeder/pom.xml
@@ -33,8 +33,8 @@
UTF-8
-5.1.1.RELEASE
-2.0.6.RELEASE
+5.1.2.RELEASE
+2.1.0.RELEASE
@@ -96,7 +96,7 @@
org.easymock
easymock
- 3.6
+ 4.0.1
test
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
index b4a2a26b43..c4b9835d07 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogEntryParseTester.java
@@ -27,9 +27,11 @@
import org.apache.ambari.logfeeder.conf.LogEntryCacheConfig;
import org.apache.ambari.logfeeder.conf.LogFeederProps;
+import
org.apache.ambari.logfeeder.manager.operations.impl.DefaultInputConfigHandler;
import org.apache.ambari.logfeeder.input.InputFileMarker;
import org.apache.ambari.logfeeder.input.InputManagerImpl;
import org.apache.ambari.logfeeder.loglevelfilter.LogLevelFilterHandler;
+import org.apache.ambari.logfeeder.manager.InputConfigManager;
import org.apache.ambari.logfeeder.output.OutputManagerImpl;
import org.apache.ambari.logfeeder.plugin.input.Input;
import org.apache.ambari.logfeeder.plugin.input.InputMarker;
@@ -89,8 +91,6 @@ public LogEntryParseTester(String logEntry, String
shipperConfig, List g
@SuppressWarnings("unchecked")
public Map parse() throws Exception {
InputConfig inputConfig = getInputConfig();
-ConfigHandler configHandler = new ConfigHandler(null);
-configHandler.setInputManager(new InputManagerImpl());
OutputManagerImpl outputManager = new OutputManagerImpl();
LogFeederProps logFeederProps = new LogFeederProps();
LogEntryCacheConfig logEntryCacheConfig = new LogEntryCacheConfig();
@@ -101,8 +101,11 @@ public LogEntryParseTester(String logEntry, String
shipperConfig, List g
LogLevelFilterHandler logLevelFilterHandler = new
LogLevelFilterHandler(null);
logLevelFilterHandler.setLogFeederProps(logFeederProps);
outputManager.setLogLevelFilterHandler(logLevelFilterHandler);
-configHandler.setOutputManager(outputManager);
-Input input = configHandler.getTestInput(inputConfig, logId);
+DefaultInputConfigHandler configHandler = new DefaultInputConfigHandler();
+InputConfigManager inputConfigManager = new InputConfigManager(
+ null, new InputManagerImpl(), outputManager,
configHandler,logFeederProps, true
+);
+Input input = inputConfigManager.getTestInput(inputConfig, logId);
input.init(logFeederProps);
final Map result = new HashMap<>();
input.getFirstFilter().init(logFeederProps);
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
b/ambari-logsearch-
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670654#comment-16670654
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229844986
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/credential/FileSecretStore.java
##
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.credential;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.File;
+import java.nio.charset.Charset;
+
+public class FileSecretStore implements SecretStore {
+
+ private static final Logger logger =
LogManager.getLogger(FileSecretStore.class);
+
+ private final String fileLocation;
+ private final String defaultSecret;
+
+ public FileSecretStore(String fileLocation, String defaultSecret) {
+this.fileLocation = fileLocation;
+this.defaultSecret = defaultSecret;
+ }
+
+ public FileSecretStore(String fileLocation) {
+this.fileLocation = fileLocation;
+this.defaultSecret = null;
+ }
+
+ @Override
+ public char[] getSecret() {
+try {
+ File pwdFile = new File(fileLocation);
+ if (!pwdFile.exists() && defaultSecret != null) {
+FileUtils.writeStringToFile(pwdFile, defaultSecret,
Charset.defaultCharset());
+return defaultSecret.toCharArray();
+ } else {
+return FileUtils.readFileToString(pwdFile,
Charset.defaultCharset()).toCharArray();
Review comment:
File content can be read directly to byte or char array That would be more
secure than readFileToString. Also defaultSecret can be a char[]
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670444#comment-16670444 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #17: AMBARI-24833. Create cloud input/output skeleton. URL: https://github.com/apache/ambari-logsearch/pull/17#issuecomment-434783497 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/27/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670305#comment-16670305 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #17: AMBARI-24833. Create cloud input/output skeleton. URL: https://github.com/apache/ambari-logsearch/pull/17#issuecomment-434743372 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/25/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670275#comment-16670275 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #17: AMBARI-24833. Create cloud input/output skeleton. URL: https://github.com/apache/ambari-logsearch/pull/17#issuecomment-434739915 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/24/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670164#comment-16670164
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229713010
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/DefaultInputConfigHandler.java
##
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.manager.operations.impl;
+
+import org.apache.ambari.logfeeder.manager.operations.InputConfigHandler;
+import org.apache.ambari.logfeeder.input.InputSimulate;
+import org.apache.ambari.logfeeder.manager.InputConfigHolder;
+import org.apache.ambari.logfeeder.plugin.common.AliasUtil;
+import org.apache.ambari.logfeeder.plugin.filter.Filter;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.ambari.logfeeder.plugin.output.Output;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.FilterDescriptor;
+import org.apache.ambari.logsearch.config.api.model.inputconfig.InputConfig;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.InputDescriptor;
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Holds input/filter/output operations in default Log Feeder mode.
+ */
+public class DefaultInputConfigHandler implements InputConfigHandler {
+
+ private static final Logger logger =
LogManager.getLogger(DefaultInputConfigHandler.class);
+
+ @Override
+ public void init(InputConfigHolder inputConfigHolder) throws Exception {
+ }
+
+ @Override
+ public void loadInputs(String serviceName, InputConfigHolder
inputConfigHolder, InputConfig inputConfig) {
+loadInputs(serviceName, inputConfigHolder);
+loadFilters(serviceName, inputConfigHolder);
+ }
+
+ @Override
+ public void assignInputsToOutputs(String serviceName, InputConfigHolder
inputConfigHolder, InputConfig config) {
+for (Input input :
inputConfigHolder.getInputManager().getInputList(serviceName)) {
+ for (Output output : inputConfigHolder.getOutputManager().getOutputs()) {
+if (input.isOutputRequired(output)) {
+ input.addOutput(output);
+}
+ }
+}
+
+// In case of simulation copies of the output are added for each
simulation instance, these must be added to the manager
+for (Output output : InputSimulate.getSimulateOutputs()) {
+ output.setLogSearchConfig(inputConfigHolder.getConfig());
+ inputConfigHolder.getOutputManager().add(output);
+}
+ }
+
+ private void loadInputs(String serviceName, InputConfigHolder
inputConfigHolder) {
+for (InputDescriptor inputDescriptor :
inputConfigHolder.getInputConfigList()) {
+ if (inputDescriptor == null) {
+continue;
+ }
+
+ String source = inputDescriptor.getSource();
+ if (StringUtils.isEmpty(source)) {
+logger.error("Input block doesn't have source element");
+continue;
+ }
+ Input input = (Input) AliasUtil.getClassInstance(source,
AliasUtil.AliasType.INPUT);
+ if (input == null) {
+logger.error("Input object could not be found");
+continue;
+ }
+ input.setType(source);
+ input.setLogType(inputDescriptor.getType());
+ input.loadConfig(inputDescriptor);
+
+ if (input.isEnabled()) {
+input.setOutputManager(inputConfigHolder.getOutputManager());
+input.setInputManager(inputConfigHolder.getInputManager());
+inputConfigHolder.getInputManager().add(serviceName, input);
+logger.info("New input object registered for service '{}': '{}'",
serviceName, input.getLogType());
+input.logConfigs();
+ } else {
+logger.info("Input is disabled. So ignoring it. " +
i
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670163#comment-16670163
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229712754
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/DefaultInputConfigHandler.java
##
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.manager.operations.impl;
+
+import org.apache.ambari.logfeeder.manager.operations.InputConfigHandler;
+import org.apache.ambari.logfeeder.input.InputSimulate;
+import org.apache.ambari.logfeeder.manager.InputConfigHolder;
+import org.apache.ambari.logfeeder.plugin.common.AliasUtil;
+import org.apache.ambari.logfeeder.plugin.filter.Filter;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.ambari.logfeeder.plugin.output.Output;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.FilterDescriptor;
+import org.apache.ambari.logsearch.config.api.model.inputconfig.InputConfig;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.InputDescriptor;
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Holds input/filter/output operations in default Log Feeder mode.
+ */
+public class DefaultInputConfigHandler implements InputConfigHandler {
+
+ private static final Logger logger =
LogManager.getLogger(DefaultInputConfigHandler.class);
+
+ @Override
+ public void init(InputConfigHolder inputConfigHolder) throws Exception {
+ }
+
+ @Override
+ public void loadInputs(String serviceName, InputConfigHolder
inputConfigHolder, InputConfig inputConfig) {
+loadInputs(serviceName, inputConfigHolder);
+loadFilters(serviceName, inputConfigHolder);
+ }
+
+ @Override
+ public void assignInputsToOutputs(String serviceName, InputConfigHolder
inputConfigHolder, InputConfig config) {
+for (Input input :
inputConfigHolder.getInputManager().getInputList(serviceName)) {
+ for (Output output : inputConfigHolder.getOutputManager().getOutputs()) {
+if (input.isOutputRequired(output)) {
+ input.addOutput(output);
+}
+ }
+}
+
+// In case of simulation copies of the output are added for each
simulation instance, these must be added to the manager
+for (Output output : InputSimulate.getSimulateOutputs()) {
+ output.setLogSearchConfig(inputConfigHolder.getConfig());
+ inputConfigHolder.getOutputManager().add(output);
+}
+ }
+
+ private void loadInputs(String serviceName, InputConfigHolder
inputConfigHolder) {
+for (InputDescriptor inputDescriptor :
inputConfigHolder.getInputConfigList()) {
+ if (inputDescriptor == null) {
+continue;
Review comment:
Should we log an Error or throw an exception when `inputDescriptor` is
`null`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JI
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670155#comment-16670155
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229709887
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/CloudStorageInputConfigHandler.java
##
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.manager.operations.impl;
+
+import org.apache.ambari.logfeeder.common.LogFeederConstants;
+import org.apache.ambari.logfeeder.conf.LogFeederMode;
+import org.apache.ambari.logfeeder.filter.FilterDummy;
+import org.apache.ambari.logfeeder.manager.operations.InputConfigHandler;
+import org.apache.ambari.logfeeder.manager.InputConfigHolder;
+import org.apache.ambari.logfeeder.plugin.common.AliasUtil;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.ambari.logfeeder.plugin.output.Output;
+import org.apache.ambari.logsearch.config.api.model.inputconfig.InputConfig;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.InputDescriptor;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.InputSocketDescriptor;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+
+/**
+ * Holds input/filter/output operations in cloud Log Feeder mode.
+ */
+public class CloudStorageInputConfigHandler implements InputConfigHandler {
+
+ private static final Logger logger =
LogManager.getLogger(CloudStorageInputConfigHandler.class);
+
+ @Override
+ public void init(InputConfigHolder inputConfigHolder) {
+logger.info("Call init of cloud input config handler...");
+ }
+
+ @Override
+ public void loadInputs(String serviceName, InputConfigHolder
inputConfigHolder, InputConfig inputConfig) {
+for (InputDescriptor inputDescriptor :
inputConfigHolder.getInputConfigList()) {
+ if (inputDescriptor == null) {
+continue;
Review comment:
Should an exception be thrown here? like
```
throw new NullPointerException("inputDescriptor can not be null!")
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670130#comment-16670130
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229701401
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
##
@@ -199,6 +199,16 @@
@Value("${" + LogFeederConstants.SOLR_URLS + ":}")
private String solrUrlsStr;
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.CLOUD_STORAGE_MODE,
+description = "Option to support sending logs to cloud storage. You can
choose between supporting only cloud storage, non-cloud storage or both",
+examples = {"default", "cloud", "hybrid"},
+defaultValue = "true",
+sources = {LogFeederConstants.CLOUD_STORAGE_MODE}
+ )
+ @Value("${" + LogFeederConstants.CLOUD_STORAGE_MODE + ":default}")
+ public String cloudStorageMode;
Review comment:
it can be, note that in the Condition classes, we will need to gather that
from a string anyway
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670057#comment-16670057
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229679354
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
##
@@ -199,6 +199,16 @@
@Value("${" + LogFeederConstants.SOLR_URLS + ":}")
private String solrUrlsStr;
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.CLOUD_STORAGE_MODE,
+description = "Option to support sending logs to cloud storage. You can
choose between supporting only cloud storage, non-cloud storage or both",
+examples = {"default", "cloud", "hybrid"},
+defaultValue = "true",
+sources = {LogFeederConstants.CLOUD_STORAGE_MODE}
+ )
+ @Value("${" + LogFeederConstants.CLOUD_STORAGE_MODE + ":default}")
+ public String cloudStorageMode;
Review comment:
Can `cloudStorageMode` have a type of `LogFeederMode` instead of `String`?
Or `LogFeederMode.fromString(mode)` should throw an informative exception in
case of an invalid value instead of falling back to `LogFeederMode.DEFAULT` like
```
throw new IllegalArgumentException("The value '" + mode + "'is not a valid
LogFeederMode!");
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670052#comment-16670052
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229679354
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
##
@@ -199,6 +199,16 @@
@Value("${" + LogFeederConstants.SOLR_URLS + ":}")
private String solrUrlsStr;
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.CLOUD_STORAGE_MODE,
+description = "Option to support sending logs to cloud storage. You can
choose between supporting only cloud storage, non-cloud storage or both",
+examples = {"default", "cloud", "hybrid"},
+defaultValue = "true",
+sources = {LogFeederConstants.CLOUD_STORAGE_MODE}
+ )
+ @Value("${" + LogFeederConstants.CLOUD_STORAGE_MODE + ":default}")
+ public String cloudStorageMode;
Review comment:
Can `cloudStorageMode` have a type of `LogFeederMode` instead of `String`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670053#comment-16670053
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229679354
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
##
@@ -199,6 +199,16 @@
@Value("${" + LogFeederConstants.SOLR_URLS + ":}")
private String solrUrlsStr;
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.CLOUD_STORAGE_MODE,
+description = "Option to support sending logs to cloud storage. You can
choose between supporting only cloud storage, non-cloud storage or both",
+examples = {"default", "cloud", "hybrid"},
+defaultValue = "true",
+sources = {LogFeederConstants.CLOUD_STORAGE_MODE}
+ )
+ @Value("${" + LogFeederConstants.CLOUD_STORAGE_MODE + ":default}")
+ public String cloudStorageMode;
Review comment:
Can `cloudStorageMode` have a type of `LogFeederMode` instead of `String`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670049#comment-16670049
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229678607
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
##
@@ -199,6 +199,16 @@
@Value("${" + LogFeederConstants.SOLR_URLS + ":}")
private String solrUrlsStr;
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.CLOUD_STORAGE_MODE,
+description = "Option to support sending logs to cloud storage. You can
choose between supporting only cloud storage, non-cloud storage or both",
+examples = {"default", "cloud", "hybrid"},
+defaultValue = "true",
Review comment:
Is `true` a valid value for `cloudStorageMode`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670045#comment-16670045
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #17: AMBARI-24833. Create cloud
input/output skeleton.
URL: https://github.com/apache/ambari-logsearch/pull/17#discussion_r229677987
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederMode.java
##
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.conf;
+
+/**
+ * Global Log Feeder modes:
+ *
+ * - default: process logs based on input / filter / output JSON configurations
+ * - cloud: process logs based on input JSON configurations and send them
directly into cloud storage (without filters)
+ * - hybrid: use both 2 above (together)
+ *
+ */
+public enum LogFeederMode {
+ DEFAULT("default"), CLOUD("cloud"), HYBRID("hybrid");
+
+ private String text;
+
+ LogFeederMode(String text) {
+this.text = text;
+ }
+
+ public String getText() {
+return this.text;
+ }
+
+ public static LogFeederMode fromString(String text) {
+for (LogFeederMode mode : LogFeederMode.values()) {
+ if (mode.text.equalsIgnoreCase(text)) {
+return mode;
+ }
+}
+return LogFeederMode.DEFAULT;
+ }
+
+ public static boolean isCloudStorage(LogFeederMode mode) {
+return mode.equals(LogFeederMode.HYBRID) ||
mode.equals(LogFeederMode.CLOUD);
Review comment:
If would be safer to check equality this way because `mode` can be null:
```
LogFeederMode.HYBRID.equals(mode)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16670023#comment-16670023 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #17: AMBARI-24833. Create cloud input/output skeleton. URL: https://github.com/apache/ambari-logsearch/pull/17 # What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. Ex: unit tests, manual tests) (If this patch involves UI changes, please attach a screen-shot; otherwise, remove this) Please review [Ambari Contributing Guide](https://cwiki.apache.org/confluence/display/AMBARI/How+to+Contribute) before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
