[jira] [Created] (AMBARI-24926) Apply user-defined configuration for Add Service request
Doroszlai, Attila created AMBARI-24926: -- Summary: Apply user-defined configuration for Add Service request Key: AMBARI-24926 URL: https://issues.apache.org/jira/browse/AMBARI-24926 Project: Ambari Issue Type: Task Components: ambari-server Reporter: Doroszlai, Attila Assignee: Doroszlai, Attila Fix For: 2.8.0 Continuing AMBARI-24917, apply any configuration specified in the request to override the stack defaults. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (AMBARI-24917) Implement complex Add Service request using default configs
[ https://issues.apache.org/jira/browse/AMBARI-24917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doroszlai, Attila reopened AMBARI-24917: > Implement complex Add Service request using default configs > --- > > Key: AMBARI-24917 > URL: https://issues.apache.org/jira/browse/AMBARI-24917 > Project: Ambari > Issue Type: Task > Components: ambari-server >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Major > Labels: pull-request-available > Fix For: trunk > > Time Spent: 0.5h > Remaining Estimate: 0h > > Continuing AMBARI-24901, create components and host components for the Add > Service request. Also create stack default configs, necessary for install > and start tasks. > User-defined and advisor-recommended configs to be implemented separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24917) Implement complex Add Service request using default configs
[ https://issues.apache.org/jira/browse/AMBARI-24917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692783#comment-16692783 ] Hudson commented on AMBARI-24917: - FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #10311 (See [https://builds.apache.org/job/Ambari-trunk-Commit/10311/]) AMBARI-24917. Implement complex Add Service request using default (github: [https://gitbox.apache.org/repos/asf?p=ambari.git&a=commit&h=94d18601e8adef6bd43d2ce1e70d3c1e4f14ae3c]) * (edit) ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Stack.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/topology/addservice/ResourceProviderAdapter.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UnitUpdater.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/topology/addservice/AddServiceOrchestrator.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostComponentResourceProvider.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceResourceProvider.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/topology/addservice/AddServiceInfo.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/api/handlers/CreateHandler.java > Implement complex Add Service request using default configs > --- > > Key: AMBARI-24917 > URL: https://issues.apache.org/jira/browse/AMBARI-24917 > Project: Ambari > Issue Type: Task > Components: ambari-server >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Continuing AMBARI-24901, create components and host components for the Add > Service request. Also create stack default configs, necessary for install > and start tasks. > User-defined and advisor-recommended configs to be implemented separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AMBARI-24917) Implement complex Add Service request using default configs
[ https://issues.apache.org/jira/browse/AMBARI-24917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doroszlai, Attila resolved AMBARI-24917. Resolution: Fixed > Implement complex Add Service request using default configs > --- > > Key: AMBARI-24917 > URL: https://issues.apache.org/jira/browse/AMBARI-24917 > Project: Ambari > Issue Type: Task > Components: ambari-server >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Major > Labels: pull-request-available > Fix For: trunk > > Time Spent: 0.5h > Remaining Estimate: 0h > > Continuing AMBARI-24901, create components and host components for the Add > Service request. Also create stack default configs, necessary for install > and start tasks. > User-defined and advisor-recommended configs to be implemented separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24881) Implement JSON parsing code to handle Add Service request
[ https://issues.apache.org/jira/browse/AMBARI-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doroszlai, Attila updated AMBARI-24881: --- Fix Version/s: (was: trunk) 2.8.0 > Implement JSON parsing code to handle Add Service request > - > > Key: AMBARI-24881 > URL: https://issues.apache.org/jira/browse/AMBARI-24881 > Project: Ambari > Issue Type: Task > Components: ambari-server >Affects Versions: 2.7.1 >Reporter: Balázs Bence Sári >Assignee: Balázs Bence Sári >Priority: Critical > Labels: pull-request-available > Fix For: 2.8.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > This JIRA tracks the work involved to implement the parsing/handling code for > the JSON request and response elements of the "Add Service" API. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AMBARI-24917) Implement complex Add Service request using default configs
[ https://issues.apache.org/jira/browse/AMBARI-24917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doroszlai, Attila resolved AMBARI-24917. Resolution: Implemented > Implement complex Add Service request using default configs > --- > > Key: AMBARI-24917 > URL: https://issues.apache.org/jira/browse/AMBARI-24917 > Project: Ambari > Issue Type: Task > Components: ambari-server >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Major > Labels: pull-request-available > Fix For: trunk > > Time Spent: 0.5h > Remaining Estimate: 0h > > Continuing AMBARI-24901, create components and host components for the Add > Service request. Also create stack default configs, necessary for install > and start tasks. > User-defined and advisor-recommended configs to be implemented separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24917) Implement complex Add Service request using default configs
[ https://issues.apache.org/jira/browse/AMBARI-24917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doroszlai, Attila updated AMBARI-24917: --- Fix Version/s: (was: trunk) 2.8.0 > Implement complex Add Service request using default configs > --- > > Key: AMBARI-24917 > URL: https://issues.apache.org/jira/browse/AMBARI-24917 > Project: Ambari > Issue Type: Task > Components: ambari-server >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Continuing AMBARI-24901, create components and host components for the Add > Service request. Also create stack default configs, necessary for install > and start tasks. > User-defined and advisor-recommended configs to be implemented separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24925) DFS Directory creation fails for Spark History
[ https://issues.apache.org/jira/browse/AMBARI-24925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-24925: - Status: Patch Available (was: Open) > DFS Directory creation fails for Spark History > -- > > Key: AMBARI-24925 > URL: https://issues.apache.org/jira/browse/AMBARI-24925 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Fix For: 2.8.0 > > Attachments: AMBARI-24925.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24916) Ambari Server setup for non embedded DB miss out on default configuration
[
https://issues.apache.org/jira/browse/AMBARI-24916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692774#comment-16692774
]
Doroszlai, Attila commented on AMBARI-24916:
Yes, the second one seems unnecessary:
https://github.com/apache/ambari/blob/93576b7c9bc25361ba1e43c9dfb08fc136667520/ambari-server/src/main/python/ambari_server/serverSetup.py#L1183-L1185
We can assume the return is intentional. You can setup several different JDBC
drivers to be used by services, and probably don't want to go over the regular
setup for each.
I think it would have been nice to if it had been added as a separate command
to avoid confusion, but at this point changing it would break existing
automation scripts.
> Ambari Server setup for non embedded DB miss out on default configuration
> -
>
> Key: AMBARI-24916
> URL: https://issues.apache.org/jira/browse/AMBARI-24916
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
>Affects Versions: 3.0.0, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.7.1, 2.7.2
>Reporter: Viraj Jasani
>Priority: Critical
> Fix For: 3.0.0, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.7.1, 2.7.2
>
>
> If we setup Ambari Server with it's own embedded postgresql DB, we get all
> other configurations correctly e.g. setup command updates home.java,
> server.os_type, server.os_family etc in ambari.properties file that can be
> used during starting Ambari Server.
>
> However, if we provide server setup command with arguments to use
> non-embedded standalone DB, server setup does not update ambari.properties
> and does not perform any further setup utility that can be leveraged during
> starting Ambari Server.
> Example:
> 1) *ambari-server setup -j \{JDK_PATH} -s*
> This command updates ambari.properties for home.java, server.os_type,
> server.os_family etc.
>
> 2) *ambari-server setup --java-home=\{JDK_PATH} --jdbc-db=postgres
> --jdbc-driver=\{JDBC_DRIVER_PATH} --databasehost=\{IP_ADDR}
> --databaseport=5432 --databasename=ambari --postgresschema=ambari
> --databaseusername=\{DB_USER} --databasepassword=\{DB_PASSWD}
> --database=postgres -s*
> This command does not update ambari.properties and we need to update it
> manually to bring up Ambari Server. Hence, automation of server setup with
> server startup is blocked due to this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMBARI-24925) DFS Directory creation fails for Spark History
[ https://issues.apache.org/jira/browse/AMBARI-24925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AMBARI-24925: Labels: pull-request-available (was: ) > DFS Directory creation fails for Spark History > -- > > Key: AMBARI-24925 > URL: https://issues.apache.org/jira/browse/AMBARI-24925 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Attachments: AMBARI-24925.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMBARI-24925) DFS Directory creation fails for Spark History
Andrew Onischuk created AMBARI-24925: Summary: DFS Directory creation fails for Spark History Key: AMBARI-24925 URL: https://issues.apache.org/jira/browse/AMBARI-24925 Project: Ambari Issue Type: Bug Reporter: Andrew Onischuk Assignee: Andrew Onischuk Fix For: 2.8.0 Attachments: AMBARI-24925.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24925) DFS Directory creation fails for Spark History
[ https://issues.apache.org/jira/browse/AMBARI-24925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-24925: - Attachment: AMBARI-24925.patch > DFS Directory creation fails for Spark History > -- > > Key: AMBARI-24925 > URL: https://issues.apache.org/jira/browse/AMBARI-24925 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Fix For: 2.8.0 > > Attachments: AMBARI-24925.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24924) hbase fails to start if fs.defaultFS is set to file:///
[ https://issues.apache.org/jira/browse/AMBARI-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AMBARI-24924: Labels: pull-request-available (was: ) > hbase fails to start if fs.defaultFS is set to file:/// > --- > > Key: AMBARI-24924 > URL: https://issues.apache.org/jira/browse/AMBARI-24924 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Attachments: AMBARI-24924.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24924) hbase fails to start if fs.defaultFS is set to file:///
[ https://issues.apache.org/jira/browse/AMBARI-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-24924: - Attachment: AMBARI-24924.patch > hbase fails to start if fs.defaultFS is set to file:/// > --- > > Key: AMBARI-24924 > URL: https://issues.apache.org/jira/browse/AMBARI-24924 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Fix For: 2.8.0 > > Attachments: AMBARI-24924.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24924) hbase fails to start if fs.defaultFS is set to file:///
[ https://issues.apache.org/jira/browse/AMBARI-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-24924: - Status: Patch Available (was: Open) > hbase fails to start if fs.defaultFS is set to file:/// > --- > > Key: AMBARI-24924 > URL: https://issues.apache.org/jira/browse/AMBARI-24924 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Fix For: 2.8.0 > > Attachments: AMBARI-24924.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMBARI-24924) hbase fails to start if fs.defaultFS is set to file:///
Andrew Onischuk created AMBARI-24924: Summary: hbase fails to start if fs.defaultFS is set to file:/// Key: AMBARI-24924 URL: https://issues.apache.org/jira/browse/AMBARI-24924 Project: Ambari Issue Type: Bug Reporter: Andrew Onischuk Assignee: Andrew Onischuk Fix For: 2.8.0 Attachments: AMBARI-24924.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24916) Ambari Server setup for non embedded DB miss out on default configuration
[
https://issues.apache.org/jira/browse/AMBARI-24916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692710#comment-16692710
]
Viraj Jasani commented on AMBARI-24916:
---
Moreover, return statement here should be appropriate only if JDBC driver setup
is intentionally not made to mix with the rest of setup, otherwise this return
statement could be a bug?
[https://github.com/apache/ambari/blob/93576b7c9bc25361ba1e43c9dfb08fc136667520/ambari-server/src/main/python/ambari_server/serverSetup.py#L1168]
> Ambari Server setup for non embedded DB miss out on default configuration
> -
>
> Key: AMBARI-24916
> URL: https://issues.apache.org/jira/browse/AMBARI-24916
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
>Affects Versions: 3.0.0, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.7.1, 2.7.2
>Reporter: Viraj Jasani
>Priority: Critical
> Fix For: 3.0.0, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.7.1, 2.7.2
>
>
> If we setup Ambari Server with it's own embedded postgresql DB, we get all
> other configurations correctly e.g. setup command updates home.java,
> server.os_type, server.os_family etc in ambari.properties file that can be
> used during starting Ambari Server.
>
> However, if we provide server setup command with arguments to use
> non-embedded standalone DB, server setup does not update ambari.properties
> and does not perform any further setup utility that can be leveraged during
> starting Ambari Server.
> Example:
> 1) *ambari-server setup -j \{JDK_PATH} -s*
> This command updates ambari.properties for home.java, server.os_type,
> server.os_family etc.
>
> 2) *ambari-server setup --java-home=\{JDK_PATH} --jdbc-db=postgres
> --jdbc-driver=\{JDBC_DRIVER_PATH} --databasehost=\{IP_ADDR}
> --databaseport=5432 --databasename=ambari --postgresschema=ambari
> --databaseusername=\{DB_USER} --databasepassword=\{DB_PASSWD}
> --database=postgres -s*
> This command does not update ambari.properties and we need to update it
> manually to bring up Ambari Server. Hence, automation of server setup with
> server startup is blocked due to this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24916) Ambari Server setup for non embedded DB miss out on default configuration
[
https://issues.apache.org/jira/browse/AMBARI-24916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692703#comment-16692703
]
Viraj Jasani commented on AMBARI-24916:
---
Hi [~adoroszlai],
Thanks for your suggestion. However, not letting JDBC driver setup mix with the
rest is intentional?
Also, I can see multiple check for JDBC properties that also seem bit confusing
on below 2 lines:
[https://github.com/apache/ambari/blob/93576b7c9bc25361ba1e43c9dfb08fc136667520/ambari-server/src/main/python/ambari_server/serverSetup.py#L1166]
[https://github.com/apache/ambari/blob/93576b7c9bc25361ba1e43c9dfb08fc136667520/ambari-server/src/main/python/ambari_server/serverSetup.py#L1184]
> Ambari Server setup for non embedded DB miss out on default configuration
> -
>
> Key: AMBARI-24916
> URL: https://issues.apache.org/jira/browse/AMBARI-24916
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
>Affects Versions: 3.0.0, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.7.1, 2.7.2
>Reporter: Viraj Jasani
>Priority: Critical
> Fix For: 3.0.0, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.7.1, 2.7.2
>
>
> If we setup Ambari Server with it's own embedded postgresql DB, we get all
> other configurations correctly e.g. setup command updates home.java,
> server.os_type, server.os_family etc in ambari.properties file that can be
> used during starting Ambari Server.
>
> However, if we provide server setup command with arguments to use
> non-embedded standalone DB, server setup does not update ambari.properties
> and does not perform any further setup utility that can be leveraged during
> starting Ambari Server.
> Example:
> 1) *ambari-server setup -j \{JDK_PATH} -s*
> This command updates ambari.properties for home.java, server.os_type,
> server.os_family etc.
>
> 2) *ambari-server setup --java-home=\{JDK_PATH} --jdbc-db=postgres
> --jdbc-driver=\{JDBC_DRIVER_PATH} --databasehost=\{IP_ADDR}
> --databaseport=5432 --databasename=ambari --postgresschema=ambari
> --databaseusername=\{DB_USER} --databasepassword=\{DB_PASSWD}
> --database=postgres -s*
> This command does not update ambari.properties and we need to update it
> manually to bring up Ambari Server. Hence, automation of server setup with
> server startup is blocked due to this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692272#comment-16692272 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #38 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/38/]) AMBARI-24833. Let logfeeder.properties override core-site configs. (#30) (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=defc854b8d1948745a07a1a6e07322670ffef340]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692258#comment-16692258 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #37 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/37/]) AMBARI-24833. Use clustername and hostname in cloud log archive dir (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=1af5b0b50442df0bb292151554a169e64372289a]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692251#comment-16692251
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #29: AMBARI-24833. Use clustername and hostname in
cloud log archive dir
URL: https://github.com/apache/ambari-logsearch/pull/29
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
index 0cfdbcc1e2..ea422b9c31 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
@@ -21,6 +21,7 @@
import org.apache.ambari.logfeeder.common.LogFeederConstants;
import org.apache.ambari.logfeeder.conf.LogFeederProps;
import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.ambari.logfeeder.util.LogFeederUtil;
import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.core.LoggerContext;
@@ -56,10 +57,11 @@ public static Logger createLogger(Input input,
LoggerContext loggerContext, LogF
String type = input.getLogType().replace(LogFeederConstants.CLOUD_PREFIX,
"");
String uniqueThreadName = input.getThread().getName();
Configuration config = loggerContext.getConfiguration();
-String destination = logFeederProps.getCloudStorageDestination().getText();
String baseDir =
logFeederProps.getRolloverConfig().getRolloverArchiveBaseDir();
-String activeLogDir = Paths.get(baseDir, destination, ACTIVE_FOLDER,
type).toFile().getAbsolutePath();
-String archiveLogDir = Paths.get(baseDir, destination, ARCHIVED_FOLDER,
type).toFile().getAbsolutePath();
+String destination = logFeederProps.getCloudStorageDestination().getText();
+String clusterHostnameBaseDir = Paths.get(baseDir, destination,
logFeederProps.getClusterName(),
LogFeederUtil.hostName).toFile().getAbsolutePath();
+String activeLogDir = Paths.get(clusterHostnameBaseDir, ACTIVE_FOLDER,
type).toFile().getAbsolutePath();
+String archiveLogDir = Paths.get(clusterHostnameBaseDir, ARCHIVED_FOLDER,
type).toFile().getAbsolutePath();
boolean useGzip = logFeederProps.getRolloverConfig().isUseGzip();
final String archiveFilePattern;
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
index af9326aed4..22c7fc1c61 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java
@@ -26,6 +26,7 @@
import org.apache.logging.log4j.Logger;
import java.io.File;
+import java.nio.file.Paths;
import java.util.Collection;
/**
@@ -74,10 +75,11 @@ public void run() {
*/
void doUpload() {
try {
- final String archiveLogDir = String.join(File.separator,
logFeederProps.getRolloverConfig().getRolloverArchiveBaseDir(), uploaderType,
"archived");
- if (new File(archiveLogDir).exists()) {
+ final File archiveLogDir =
Paths.get(logFeederProps.getRolloverConfig().getRolloverArchiveBaseDir(),
+uploaderType, clusterName, hostName, "archived").toFile();
+ if (archiveLogDir.exists()) {
String[] extensions = {"log", "json", "gz"};
-Collection filesToUpload = FileUtils.listFiles(new
File(archiveLogDir), extensions, true);
+Collection filesToUpload = FileUtils.listFiles(archiveLogDir,
extensions, true);
if (filesToUpload.isEmpty()) {
logger.debug("Not found any files to upload.");
} else {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Comp
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692252#comment-16692252
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #30: AMBARI-24833. Let logfeeder.properties
override core-site configs.
URL: https://github.com/apache/ambari-logsearch/pull/30
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java
index 5405a1051c..df1b8a55b2 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSS3UploadClient.java
@@ -57,6 +57,7 @@ public void init(LogFeederProps logFeederProps) {
conf.set("fs.s3a.endpoint", s3OutputConfig.getEndpoint());
conf.set("fs.s3a.path.style.access",
String.valueOf(s3OutputConfig.isPathStyleAccess()));
conf.set("fs.s3a.multiobjectdelete.enable",
String.valueOf(s3OutputConfig.isMultiobjectDeleteEnable()));
+LogFeederHDFSUtil.overrideFileSystemConfigs(logFeederProps, conf);
this.fs = LogFeederHDFSUtil.buildFileSystem(conf);
}
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
index 421c4c5cb7..7e1b471de0 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java
@@ -84,6 +84,7 @@ public void init(LogFeederProps logFeederProps) {
}
}
logger.info("HDFS client - will use '{}' permission for uploaded files",
hdfsOutputConfig.getHdfsFilePermissions());
+LogFeederHDFSUtil.overrideFileSystemConfigs(logFeederProps, configuration);
this.fs = LogFeederHDFSUtil.buildFileSystem(configuration);
}
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java
index 3549e04af6..b741229a13 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/LogFeederHDFSUtil.java
@@ -19,7 +19,10 @@
package org.apache.ambari.logfeeder.util;
import java.io.IOException;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.ambari.logfeeder.conf.LogFeederProps;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
@@ -95,4 +98,20 @@ public static void closeFileSystem(FileSystem fileSystem) {
}
}
}
+
+ /**
+ * Override Hadoop configuration object based on logfeeder.properties
configurations (with keys that starts with "fs." or "hadoop.*")
+ * @param logFeederProps global property holder
+ * @param configuration hadoop configuration holder
+ */
+ public static void overrideFileSystemConfigs(LogFeederProps logFeederProps,
Configuration configuration) {
+Properties properties = logFeederProps.getProperties();
+for (Map.Entry prop : properties.entrySet()) {
+ String propertyName = prop.getKey().toString();
+ if (propertyName.startsWith("fs.")) {
+logger.info("Override {} configuration (by logfeeder.properties)",
propertyName);
+configuration.set(propertyName, prop.getValue().toString());
+ }
+}
+ }
}
\ No newline at end of file
diff --git a/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
b/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
index 45c05f39cf..2537913bbe 100644
--- a/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
+++ b/ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties
@@ -68,3 +68,5 @@ logfeeder.s3.access.key=MyAccessKey
logfeeder.s3.object.acl=public-read
logfeeder.s3.path.style.access=true
logfeeder.s3.multiobject.delete.enable=false
+
+fs.s3a.acl.default=PublicReadWrite
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692249#comment-16692249 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #30: AMBARI-24833. Let logfeeder.properties override core-site configs. URL: https://github.com/apache/ambari-logsearch/pull/30#issuecomment-440036535 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/61/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692235#comment-16692235 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #30: AMBARI-24833. Let logfeeder.properties override core-site configs. URL: https://github.com/apache/ambari-logsearch/pull/30 # What changes were proposed in this pull request? core site configs can be override by logfeeder.properties. It can be useful if no core-site.xml available on a host ## How was this patch tested? FT: manually This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24923) Create tproxy-configuration category in Ambari Configurations data
[
https://issues.apache.org/jira/browse/AMBARI-24923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Levas updated AMBARI-24923:
--
Fix Version/s: (was: ambari-2.8)
2.8.0
> Create tproxy-configuration category in Ambari Configurations data
> ---
>
> Key: AMBARI-24923
> URL: https://issues.apache.org/jira/browse/AMBARI-24923
> Project: Ambari
> Issue Type: Task
> Components: ambari-server
>Affects Versions: 2.8.0
>Reporter: Robert Levas
>Assignee: Robert Levas
>Priority: Major
> Labels: tproxy
> Fix For: 2.8.0
>
>
> Create tproxy-configuration category in Ambari Configurations data with the
> following properties:
> * {{ambari.tproxy.authentication.enabled}}
> ** Determines whether to allow trusted proxy authentication when logging into
> Ambari
> ** {{true}} | {{false}}
> * {{ambari.tproxy.proxyuser.$username.hosts}}
> ** List of hosts from which trusted-proxy user ‘$username’ can connect from
> ** {{\*}} | {{c7401.ambari.apache.org}} | {{10.42.80.64,10.42.80.65}} |
> {{10.222.0.0/16,10.113.221.221}}
> * {{ambari.tproxy.proxyuser.$username.users}}
> ** List of users which the trusted-proxy user ‘$username’ can proxy for
> ** {{\*}} | {{user1,user2}}
> * {{ambari.tproxy.proxyuser.$username.groups}}
> ** List of user-groups which trusted-proxy user ‘$username’ can proxy for
> ** {{\*}} | {{group1,group2}}
> Note: {{$username}} is variable, declaring the values for a particular proxy
> user. For example "knox".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMBARI-24923) Create tproxy-configuration category in Ambari Configurations data
[
https://issues.apache.org/jira/browse/AMBARI-24923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Levas updated AMBARI-24923:
--
Affects Version/s: (was: ambari-2.8)
2.8.0
> Create tproxy-configuration category in Ambari Configurations data
> ---
>
> Key: AMBARI-24923
> URL: https://issues.apache.org/jira/browse/AMBARI-24923
> Project: Ambari
> Issue Type: Task
> Components: ambari-server
>Affects Versions: 2.8.0
>Reporter: Robert Levas
>Assignee: Robert Levas
>Priority: Major
> Labels: tproxy
> Fix For: 2.8.0
>
>
> Create tproxy-configuration category in Ambari Configurations data with the
> following properties:
> * {{ambari.tproxy.authentication.enabled}}
> ** Determines whether to allow trusted proxy authentication when logging into
> Ambari
> ** {{true}} | {{false}}
> * {{ambari.tproxy.proxyuser.$username.hosts}}
> ** List of hosts from which trusted-proxy user ‘$username’ can connect from
> ** {{\*}} | {{c7401.ambari.apache.org}} | {{10.42.80.64,10.42.80.65}} |
> {{10.222.0.0/16,10.113.221.221}}
> * {{ambari.tproxy.proxyuser.$username.users}}
> ** List of users which the trusted-proxy user ‘$username’ can proxy for
> ** {{\*}} | {{user1,user2}}
> * {{ambari.tproxy.proxyuser.$username.groups}}
> ** List of user-groups which trusted-proxy user ‘$username’ can proxy for
> ** {{\*}} | {{group1,group2}}
> Note: {{$username}} is variable, declaring the values for a particular proxy
> user. For example "knox".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AMBARI-24923) Create tproxy-configuration category in Ambari Configurations data
Robert Levas created AMBARI-24923:
-
Summary: Create tproxy-configuration category in Ambari
Configurations data
Key: AMBARI-24923
URL: https://issues.apache.org/jira/browse/AMBARI-24923
Project: Ambari
Issue Type: Task
Components: ambari-server
Affects Versions: ambari-2.8
Reporter: Robert Levas
Assignee: Robert Levas
Fix For: ambari-2.8
Create tproxy-configuration category in Ambari Configurations data with the
following properties:
* {{ambari.tproxy.authentication.enabled}}
** Determines whether to allow trusted proxy authentication when logging into
Ambari
** {{true}} | {{false}}
* {{ambari.tproxy.proxyuser.$username.hosts}}
** List of hosts from which trusted-proxy user ‘$username’ can connect from
** {{\*}} | {{c7401.ambari.apache.org}} | {{10.42.80.64,10.42.80.65}} |
{{10.222.0.0/16,10.113.221.221}}
* {{ambari.tproxy.proxyuser.$username.users}}
** List of users which the trusted-proxy user ‘$username’ can proxy for
** {{\*}} | {{user1,user2}}
* {{ambari.tproxy.proxyuser.$username.groups}}
** List of user-groups which trusted-proxy user ‘$username’ can proxy for
** {{\*}} | {{group1,group2}}
Note: {{$username}} is variable, declaring the values for a particular proxy
user. For example "knox".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24916) Ambari Server setup for non embedded DB miss out on default configuration
[
https://issues.apache.org/jira/browse/AMBARI-24916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692044#comment-16692044
]
Doroszlai, Attila commented on AMBARI-24916:
Hi [~vjasani],
Please execute {{ambari-server setup}} in 2 steps:
{code}
ambari-server setup --java-home={JDK_PATH} --databasehost={IP_ADDR}
--databaseport=5432 --databasename=ambari --postgresschema=ambari
--databaseusername={DB_USER} --databasepassword={DB_PASSWD} --database=postgres
-s
ambari-server setup --jdbc-db=postgres --jdbc-driver={JDBC_DRIVER_PATH}
{code}
For some reason the JDBC driver setup does not mix with the rest (maybe because
one can setup several drivers independently).
> Ambari Server setup for non embedded DB miss out on default configuration
> -
>
> Key: AMBARI-24916
> URL: https://issues.apache.org/jira/browse/AMBARI-24916
> Project: Ambari
> Issue Type: Bug
> Components: ambari-server
>Affects Versions: 3.0.0, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.7.1, 2.7.2
>Reporter: Viraj Jasani
>Priority: Critical
> Fix For: 3.0.0, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.7.1, 2.7.2
>
>
> If we setup Ambari Server with it's own embedded postgresql DB, we get all
> other configurations correctly e.g. setup command updates home.java,
> server.os_type, server.os_family etc in ambari.properties file that can be
> used during starting Ambari Server.
>
> However, if we provide server setup command with arguments to use
> non-embedded standalone DB, server setup does not update ambari.properties
> and does not perform any further setup utility that can be leveraged during
> starting Ambari Server.
> Example:
> 1) *ambari-server setup -j \{JDK_PATH} -s*
> This command updates ambari.properties for home.java, server.os_type,
> server.os_family etc.
>
> 2) *ambari-server setup --java-home=\{JDK_PATH} --jdbc-db=postgres
> --jdbc-driver=\{JDBC_DRIVER_PATH} --databasehost=\{IP_ADDR}
> --databaseport=5432 --databasename=ambari --postgresschema=ambari
> --databaseusername=\{DB_USER} --databasepassword=\{DB_PASSWD}
> --database=postgres -s*
> This command does not update ambari.properties and we need to update it
> manually to bring up Ambari Server. Hence, automation of server setup with
> server startup is blocked due to this bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24919) Allow Namenode HA setup with external namenodes in BlueprintConfigurationProcessor
[ https://issues.apache.org/jira/browse/AMBARI-24919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691934#comment-16691934 ] Hudson commented on AMBARI-24919: - SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #10310 (See [https://builds.apache.org/job/Ambari-trunk-Commit/10310/]) AMBARI-24919 external Namenode HA (benyoka) (#2625) (github: [https://gitbox.apache.org/repos/asf?p=ambari.git&a=commit&h=93576b7c9bc25361ba1e43c9dfb08fc136667520]) * (edit) ambari-server/src/test/java/org/apache/ambari/server/topology/ClusterTopologyImplTest.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/topology/validators/TopologyValidatorFactory.java * (edit) ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopology.java * (edit) ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java * (add) ambari-server/src/main/java/org/apache/ambari/server/topology/validators/NameNodeHaValidator.java * (add) ambari-server/src/test/java/org/apache/ambari/server/topology/validators/NameNodeHaValidatorTest.java > Allow Namenode HA setup with external namenodes in > BlueprintConfigurationProcessor > -- > > Key: AMBARI-24919 > URL: https://issues.apache.org/jira/browse/AMBARI-24919 > Project: Ambari > Issue Type: Task >Affects Versions: 2.7.2 >Reporter: Balázs Bence Sári >Assignee: Balázs Bence Sári >Priority: Critical > Labels: pull-request-available > Fix For: trunk > > Time Spent: 1h 10m > Remaining Estimate: 0h > > There is an increasing demand for clusters with some of the components being > managed externally to Ambari. > This ticket addresses the problem where the cluster contains a HDFS_CLIENT > but server HDFS server components are external and NAMENODE is HA. > In this cases, cluster template validation failed as there were 0 name nodes > which is less than the expected minimum of two. > Going forward the following setup should pass validation: > - Namenode HA is enabled > - There are 0 namenodes > - all namenode dfs rpc addresses point to an external fqdn. > This will give sufficent protection against accidentally omitted namenodes, > however will enable clusters with external namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AMBARI-24919) Allow Namenode HA setup with external namenodes in BlueprintConfigurationProcessor
[ https://issues.apache.org/jira/browse/AMBARI-24919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balázs Bence Sári resolved AMBARI-24919. Resolution: Fixed > Allow Namenode HA setup with external namenodes in > BlueprintConfigurationProcessor > -- > > Key: AMBARI-24919 > URL: https://issues.apache.org/jira/browse/AMBARI-24919 > Project: Ambari > Issue Type: Task >Affects Versions: 2.7.2 >Reporter: Balázs Bence Sári >Assignee: Balázs Bence Sári >Priority: Critical > Labels: pull-request-available > Fix For: trunk > > Time Spent: 1h 10m > Remaining Estimate: 0h > > There is an increasing demand for clusters with some of the components being > managed externally to Ambari. > This ticket addresses the problem where the cluster contains a HDFS_CLIENT > but server HDFS server components are external and NAMENODE is HA. > In this cases, cluster template validation failed as there were 0 name nodes > which is less than the expected minimum of two. > Going forward the following setup should pass validation: > - Namenode HA is enabled > - There are 0 namenodes > - all namenode dfs rpc addresses point to an external fqdn. > This will give sufficent protection against accidentally omitted namenodes, > however will enable clusters with external namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691894#comment-16691894 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #29: AMBARI-24833. Use clustername and hostname in cloud log archive dir URL: https://github.com/apache/ambari-logsearch/pull/29 # What changes were proposed in this pull request? Use hostname + cluster name as well in archive dir (to make it work properly with mounts) ## How was this patch tested? FT manually with docker env please review @g-boros This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691899#comment-16691899 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #29: AMBARI-24833. Use clustername and hostname in cloud log archive dir URL: https://github.com/apache/ambari-logsearch/pull/29#issuecomment-439942525 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/60/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24913) Update ldap-configuration category in Ambari Configurations data to contain properties to aid in automated LDAP configuration
[
https://issues.apache.org/jira/browse/AMBARI-24913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691843#comment-16691843
]
Hudson commented on AMBARI-24913:
-
SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #10309 (See
[https://builds.apache.org/job/Ambari-trunk-Commit/10309/])
AMBARI-24913. New LDAP related properties to indicate if Ambari should (github:
[https://gitbox.apache.org/repos/asf?p=ambari.git&a=commit&h=153d5f96cf1c1c68084d308bb98314b3d5231153])
* (edit) ambari-server/src/main/resources/stacks/ambari_configuration.py
* (edit)
ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog270Test.java
* (edit)
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog270.java
* (edit) ambari-server/src/test/python/TestAmbariConfiguration.py
* (edit)
ambari-server/src/main/java/org/apache/ambari/server/ldap/domain/AmbariLdapConfiguration.java
* (edit)
ambari-server/src/main/java/org/apache/ambari/server/configuration/AmbariServerConfigurationKey.java
* (edit)
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AmbariServerLDAPConfigurationHandler.java
> Update ldap-configuration category in Ambari Configurations data to contain
> properties to aid in automated LDAP configuration
> -
>
> Key: AMBARI-24913
> URL: https://issues.apache.org/jira/browse/AMBARI-24913
> Project: Ambari
> Issue Type: Task
> Components: ambari-server, security
>Affects Versions: 2.7.0
>Reporter: Sandor Molnar
>Assignee: Sandor Molnar
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.8.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Update {{ldap-configuration}} category in Ambari Configurations data to
> contain properties to aid in automated LDAP configuration:
> * {{ambari.ldap.manage_services}}
> ** This property is used to indicate whether Ambari is to manage relevant
> services' LDAP configurations or not ("true" | "false")
> * {{ambari.ldap.enabled_services}}
> ** This property is used to declare what services are expected to be
> configured for LDAP and is expected to be a comma-delimited list of services
> or "{{*}}" to indicate all services.
> Examples:
> {code:java|title=All services}
> "ambari.ldap.manage_services":"true",
> "ambari.ldap.enabled_services":"*"
> {code}
> {code:java|title=Only Ranger}
> "ambari.ldap.manage_services":"true",
> "ambari.ldap.enabled_services":"Ranger"
> {code}
> {code:java|title=Ranger, and Atlas}
> "ambari.ldap.manage_services":"true",
> "ambari.ldap.enabled_services":"Ranger, Atlas"
> {code}
> {code:java|title=Do not manage services}
> "ambari.ldap.manage_services":"false",
> "ambari.ldap.enabled_services":""
> {code}
> Each service in the set of services should have indicated it supports LDAP
> (see BUG-114409) else it will silently be ignored.
> This value should be set via Ambari's REST API or a Blueprint.
> Upon setting this value via the Ambari REST API, it is expected that internal
> logic will be triggered to ensure the relevant services in the list are
> configured for LDAP or not configured for LDAP as the case may be.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24921) If service does not have client service component created then "Run Service Check" option should be made hidden
[ https://issues.apache.org/jira/browse/AMBARI-24921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691759#comment-16691759 ] Hudson commented on AMBARI-24921: - FAILURE: Integrated in Jenkins build Ambari-trunk-Commit #10308 (See [https://builds.apache.org/job/Ambari-trunk-Commit/10308/]) AMBARI-24921 If service does not have client service component created (1963907+atkach: [https://gitbox.apache.org/repos/asf?p=ambari.git&a=commit&h=f4bd455ed5eb67df592169201455e95c614a69d6]) * (edit) ambari-web/test/views/main/service/item_test.js * (edit) ambari-web/app/views/main/service/item.js > If service does not have client service component created then "Run Service > Check" option should be made hidden > --- > > Key: AMBARI-24921 > URL: https://issues.apache.org/jira/browse/AMBARI-24921 > Project: Ambari > Issue Type: Task > Components: ambari-web >Affects Versions: 2.7.0 >Reporter: Andrii Tkach >Assignee: Andrii Tkach >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Time Spent: 50m > Remaining Estimate: 0h > > If service does not have client service component created then "Run Service > Check" option should be made hidden -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691756#comment-16691756
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #28: AMBARI-24833. Use seconds instead of minutes
for cloud log threshold
URL: https://github.com/apache/ambari-logsearch/pull/28
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
index a15ac7468a..1b0aa534d6 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
@@ -117,7 +117,7 @@
public static final String CLOUD_STORAGE_BASE_PATH =
"logfeeder.cloud.storage.base.path";
public static final String CLOUD_ROLLOVER_ARCHIVE_LOCATION =
"logfeeder.cloud.rollover.archive.base.dir";
- public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_MIN =
"logfeeder.cloud.rollover.threshold.min";
+ public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_SECONDS =
"logfeeder.cloud.rollover.threshold.min";
public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE =
"logfeeder.cloud.rollover.threshold.size";
public static final String CLOUD_ROLLOVER_MAX_BACKUP_FILES =
"logfeeder.cloud.rollover.max.files";
public static final String CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE_UNIT =
"logfeeder.cloud.rollover.threshold.size.unit";
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java
index 282792ae86..bdc908bd5e 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/RolloverConfig.java
@@ -37,14 +37,14 @@
private String rolloverArchiveBaseDir;
@LogSearchPropertyDescription(
-name = LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_MIN,
-description = "Rollover cloud log files after an interval (minutes)",
-examples = {"1"},
-defaultValue = "60",
+name = LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SECONDS,
+description = "Rollover cloud log files after an interval (seconds)",
+examples = {"60"},
+defaultValue = "3600",
sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
)
- @Value("${"+ LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_MIN + ":60}")
- private int rolloverThresholdTimeMins;
+ @Value("${"+ LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SECONDS +
":3600}")
+ private int rolloverThresholdTimeSeconds;
@LogSearchPropertyDescription(
name = LogFeederConstants.CLOUD_ROLLOVER_THRESHOLD_TIME_SIZE,
@@ -116,12 +116,12 @@
@Value("${"+ LogFeederConstants.CLOUD_ROLLOVER_ON_STARTUP + ":false}")
private boolean rolloverOnStartup;
- public int getRolloverThresholdTimeMins() {
-return rolloverThresholdTimeMins;
+ public int getRolloverThresholdTimeSeconds() {
+return rolloverThresholdTimeSeconds;
}
- public void setRolloverThresholdTimeMins(int rolloverThresholdTimeMins) {
-this.rolloverThresholdTimeMins = rolloverThresholdTimeMins;
+ public void setRolloverThresholdTimeSeconds(int
rolloverThresholdTimeSeconds) {
+this.rolloverThresholdTimeSeconds = rolloverThresholdTimeSeconds;
}
public Integer getRolloverMaxBackupFiles() {
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
index 0cfdbcc1e2..d39349f4d1 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java
@@ -78,7 +78,7 @@ public static Logger createLogger(Input input, LoggerContext
loggerContext, LogF
String rolloverSize =
logFeederProps.getRolloverConfig().getRolloverSize().toString() +
logFeederProps.getRolloverConfig().getRolloverSizeFormat();
SizeBasedTriggeringPolicy sizeBasedTriggeringPolicy =
SizeBasedTriggeringPolicy.createPolicy(rolloverSize);
CustomTimeBasedTriggeringPolicy customTimeBasedTriggeringPolicy =
CustomTimeBasedTriggeringPolicy
-
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691755#comment-16691755 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere commented on issue #28: AMBARI-24833. Use seconds instead of minutes for cloud log threshold URL: https://github.com/apache/ambari-logsearch/pull/28#issuecomment-439908578 i will close this, let's keep as minutes This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AMBARI-24921) If service does not have client service component created then "Run Service Check" option should be made hidden
[ https://issues.apache.org/jira/browse/AMBARI-24921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrii Tkach resolved AMBARI-24921. --- Resolution: Fixed > If service does not have client service component created then "Run Service > Check" option should be made hidden > --- > > Key: AMBARI-24921 > URL: https://issues.apache.org/jira/browse/AMBARI-24921 > Project: Ambari > Issue Type: Task > Components: ambari-web >Affects Versions: 2.7.0 >Reporter: Andrii Tkach >Assignee: Andrii Tkach >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Time Spent: 50m > Remaining Estimate: 0h > > If service does not have client service component created then "Run Service > Check" option should be made hidden -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24921) If service does not have client service component created then "Run Service Check" option should be made hidden
[ https://issues.apache.org/jira/browse/AMBARI-24921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691724#comment-16691724 ] Andrii Tkach commented on AMBARI-24921: --- committed to trunk > If service does not have client service component created then "Run Service > Check" option should be made hidden > --- > > Key: AMBARI-24921 > URL: https://issues.apache.org/jira/browse/AMBARI-24921 > Project: Ambari > Issue Type: Task > Components: ambari-web >Affects Versions: 2.7.0 >Reporter: Andrii Tkach >Assignee: Andrii Tkach >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Time Spent: 50m > Remaining Estimate: 0h > > If service does not have client service component created then "Run Service > Check" option should be made hidden -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691711#comment-16691711 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #28: AMBARI-24833. Use seconds instead of minutes for cloud log threshold URL: https://github.com/apache/ambari-logsearch/pull/28#issuecomment-439897680 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/59/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691700#comment-16691700 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #28: AMBARI-24833. Use seconds instead of minutes for cloud log threshold URL: https://github.com/apache/ambari-logsearch/pull/28 # What changes were proposed in this pull request? Use seconds instead of minutes for cloud log threshold ## How was this patch tested? waiting for UTs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691685#comment-16691685 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #27: AMBARI-24833. HDFS client kerberos support + small fixes URL: https://github.com/apache/ambari-logsearch/pull/27#issuecomment-43945 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/58/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24917) Implement complex Add Service request using default configs
[ https://issues.apache.org/jira/browse/AMBARI-24917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AMBARI-24917: Labels: pull-request-available (was: ) > Implement complex Add Service request using default configs > --- > > Key: AMBARI-24917 > URL: https://issues.apache.org/jira/browse/AMBARI-24917 > Project: Ambari > Issue Type: Task > Components: ambari-server >Reporter: Doroszlai, Attila >Assignee: Doroszlai, Attila >Priority: Major > Labels: pull-request-available > Fix For: trunk > > > Continuing AMBARI-24901, create components and host components for the Add > Service request. Also create stack default configs, necessary for install > and start tasks. > User-defined and advisor-recommended configs to be implemented separately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691688#comment-16691688 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #36 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/36/]) AMBARI-24833. HDFS client kerberos support + small fixes (#27) (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=c94d3b20576a179f7a4e04491e423c77a3976ce5]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/CloudStorageInputConfigHandler.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputLineEnricher.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/ExternalHdfsOutputConfig.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/DefaultInputConfigHandler.java * (delete) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/ExternalHDFSUploadClient.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/HdfsOutputConfig.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/HDFSUploadClient.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/upload/UploadClientFactory.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/AbstractInputConfigHandler.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691686#comment-16691686
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #27: AMBARI-24833. HDFS client kerberos support +
small fixes
URL: https://github.com/apache/ambari-logsearch/pull/27
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
index f9ef32d688..a15ac7468a 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
@@ -132,6 +132,8 @@
public static final String HDFS_PORT = "logfeeder.hdfs.port";
public static final String HDFS_FILE_PERMISSIONS =
"logfeeder.hdfs.file.permissions";
public static final String HDFS_KERBEROS = "logfeeder.hdfs.kerberos";
+ public static final String HDFS_KERBEROS_KEYTAB = "logfeeder.hdfs.keytab";
+ public static final String HDFS_KERBEROS_PRINCIPAL =
"logfeeder.hdfs.principal";
public static final String S3_ENDPOINT = "logfeeder.s3.endpoint";
public static final String S3_ENDPOINT_DEFAULT = "https://s3.amazonaws.com";;
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
index f2eb6c741e..b6ab4c7342 100644
---
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
+++
b/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java
@@ -19,7 +19,7 @@
package org.apache.ambari.logfeeder.conf;
import org.apache.ambari.logfeeder.common.LogFeederConstants;
-import org.apache.ambari.logfeeder.conf.output.ExternalHdfsOutputConfig;
+import org.apache.ambari.logfeeder.conf.output.HdfsOutputConfig;
import org.apache.ambari.logfeeder.conf.output.RolloverConfig;
import org.apache.ambari.logfeeder.conf.output.S3OutputConfig;
import org.apache.ambari.logfeeder.plugin.common.LogFeederProperties;
@@ -53,7 +53,7 @@
private S3OutputConfig s3OutputConfig;
@Inject
- private ExternalHdfsOutputConfig hdfsOutputConfig;
+ private HdfsOutputConfig hdfsOutputConfig;
private Properties properties;
@@ -258,7 +258,7 @@
defaultValue = "false",
sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
)
- @Value("${" + LogFeederConstants.CLOUD_STORAGE_USE_HDFS_CLIENT + ":false}")
+ @Value("${" + LogFeederConstants.CLOUD_STORAGE_USE_HDFS_CLIENT + ":true}")
private boolean useCloudHdfsClient;
@LogSearchPropertyDescription(
@@ -280,15 +280,6 @@
@Value("${" + LogFeederConstants.CLOUD_STORAGE_BASE_PATH + ":}")
private String cloudBasePath;
- @LogSearchPropertyDescription(
-name = LogFeederConstants.HDFS_USER,
-description = "Overrides HADOOP_USER_NAME variable at runtime",
-examples = {"hdfs"},
-sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
- )
- @Value("${"+ LogFeederConstants.HDFS_USER + ":}")
- private String logfeederHdfsUser;
-
@LogSearchPropertyDescription(
name = LogFeederConstants.CLOUD_STORAGE_USE_FILTERS,
description = "Use filters for inputs (with filters the output format will
be JSON)",
@@ -460,7 +451,7 @@ public void setCloudStorageMode(LogFeederMode
cloudStorageMode) {
this.cloudStorageMode = cloudStorageMode;
}
- public ExternalHdfsOutputConfig getHdfsOutputConfig() {
+ public HdfsOutputConfig getHdfsOutputConfig() {
return hdfsOutputConfig;
}
@@ -480,7 +471,7 @@ public void setRolloverConfig(RolloverConfig
rolloverConfig) {
this.rolloverConfig = rolloverConfig;
}
- public void setHdfsOutputConfig(ExternalHdfsOutputConfig hdfsOutputConfig) {
+ public void setHdfsOutputConfig(HdfsOutputConfig hdfsOutputConfig) {
this.hdfsOutputConfig = hdfsOutputConfig;
}
@@ -512,14 +503,6 @@ public boolean isUseCloudHdfsClient() {
return useCloudHdfsClient;
}
- public String getLogfeederHdfsUser() {
-return logfeederHdfsUser;
- }
-
- public void setLogfeederHdfsUser(String logfeederHdfsUser) {
-this.logfeederHdfsUser = logfeederHdfsUser;
- }
-
public void setUseCloudHdfsClient(boolean useCloudHdfsClient) {
this.useCloudHdfsClient = useCloudHdfsClient;
}
diff --git
a/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/ExternalHdfsOutputConfig.jav
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691678#comment-16691678
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #27: AMBARI-24833. HDFS client
kerberos support + small fixes
URL: https://github.com/apache/ambari-logsearch/pull/27#discussion_r234610512
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/output/HdfsOutputConfig.java
##
@@ -62,7 +71,27 @@
sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
)
@Value("${"+ LogFeederConstants.HDFS_KERBEROS + ":false}")
- private boolean secure;
+ private boolean hdfsKerberos;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.HDFS_KERBEROS_KEYTAB,
+description = "Kerberos keytab location for Log Feeder for communicating
with secure HDFS. ",
+examples = {"/etc/security/keytabs/mykeytab.keytab"},
+defaultValue = "/etc/security/keytabs/logfeeder.service.keytab",
+sources = {LogFeederConstants.LOGFEEDER_PROPERTIES_FILE}
+ )
+ @Value("${"+ LogFeederConstants.HDFS_KERBEROS_KEYTAB +
":/etc/security/keytabs/logfeeder.service.keytab}")
+ private String keytab;
+
+ @LogSearchPropertyDescription(
+name = LogFeederConstants.HDFS_KERBEROS_PRINCIPAL,
+description = "Kerberos keytab location for Log Feeder for communicating
with secure HDFS. ",
Review comment:
This is the same as in line 78
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMBARI-24922) No need to create test jar if tests are skipped
[
https://issues.apache.org/jira/browse/AMBARI-24922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doroszlai, Attila updated AMBARI-24922:
---
Status: Patch Available (was: Open)
> No need to create test jar if tests are skipped
> ---
>
> Key: AMBARI-24922
> URL: https://issues.apache.org/jira/browse/AMBARI-24922
> Project: Ambari
> Issue Type: Improvement
> Components: ambari-server
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
> Labels: pull-request-available
> Fix For: trunk
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Some test-related tasks ({{create-sample-upgrade-check-jar}},
> {{generate-test-oozie2-checks-dir}},
> {{generate-test-oozie2-server-actions-dir}}) can be skipped during build if
> tests are skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMBARI-24922) No need to create test jar if tests are skipped
[
https://issues.apache.org/jira/browse/AMBARI-24922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AMBARI-24922:
Labels: pull-request-available (was: )
> No need to create test jar if tests are skipped
> ---
>
> Key: AMBARI-24922
> URL: https://issues.apache.org/jira/browse/AMBARI-24922
> Project: Ambari
> Issue Type: Improvement
> Components: ambari-server
>Reporter: Doroszlai, Attila
>Assignee: Doroszlai, Attila
>Priority: Minor
> Labels: pull-request-available
> Fix For: trunk
>
>
> Some test-related tasks ({{create-sample-upgrade-check-jar}},
> {{generate-test-oozie2-checks-dir}},
> {{generate-test-oozie2-server-actions-dir}}) can be skipped during build if
> tests are skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24912) Infra Manager: scheduled job fails with dateparse exception
[ https://issues.apache.org/jira/browse/AMBARI-24912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691653#comment-16691653 ] Hudson commented on AMBARI-24912: - SUCCESS: Integrated in Jenkins build Ambari-Infra-master-Commit #19 (See [https://builds.apache.org/job/Ambari-Infra-master-Commit/19/]) AMBARI-24912 - Infra Manager: scheduled job fails with dateparse (github: [https://gitbox.apache.org/repos/asf?p=ambari-infra.git&a=commit&h=c818680578cb686460d0a6fd398bb4b990cd47d8]) * (edit) ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java > Infra Manager: scheduled job fails with dateparse exception > --- > > Key: AMBARI-24912 > URL: https://issues.apache.org/jira/browse/AMBARI-24912 > Project: Ambari > Issue Type: Bug > Components: ambari-infra >Affects Versions: 2.8.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24912) Infra Manager: scheduled job fails with dateparse exception
[
https://issues.apache.org/jira/browse/AMBARI-24912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691652#comment-16691652
]
ASF GitHub Bot commented on AMBARI-24912:
-
kasakrisz closed pull request #16: AMBARI-24912 - Infra Manager: scheduled job
fails with dateparse exception
URL: https://github.com/apache/ambari-infra/pull/16
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java
b/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java
index f0563f95..40fa19b4 100644
---
a/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java
+++
b/ambari-infra-manager/src/main/java/org/apache/ambari/infra/job/archive/SolrQueryBuilder.java
@@ -25,6 +25,7 @@
import java.time.Duration;
import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
import java.util.HashMap;
import java.util.Map;
@@ -33,7 +34,7 @@
public class SolrQueryBuilder {
public static String computeEnd(String end, Duration ttl) {
-return computeEnd(end, OffsetDateTime.now(), ttl);
+return computeEnd(end, OffsetDateTime.now(ZoneOffset.UTC), ttl);
}
public static String computeEnd(String end, OffsetDateTime now, Duration
ttl) {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Infra Manager: scheduled job fails with dateparse exception
> ---
>
> Key: AMBARI-24912
> URL: https://issues.apache.org/jira/browse/AMBARI-24912
> Project: Ambari
> Issue Type: Bug
> Components: ambari-infra
>Affects Versions: 2.8.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AMBARI-24922) No need to create test jar if tests are skipped
Doroszlai, Attila created AMBARI-24922:
--
Summary: No need to create test jar if tests are skipped
Key: AMBARI-24922
URL: https://issues.apache.org/jira/browse/AMBARI-24922
Project: Ambari
Issue Type: Improvement
Components: ambari-server
Reporter: Doroszlai, Attila
Assignee: Doroszlai, Attila
Fix For: trunk
Some test-related tasks ({{create-sample-upgrade-check-jar}},
{{generate-test-oozie2-checks-dir}},
{{generate-test-oozie2-server-actions-dir}}) can be skipped during build if
tests are skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AMBARI-24913) Update ldap-configuration category in Ambari Configurations data to contain properties to aid in automated LDAP configuration
[
https://issues.apache.org/jira/browse/AMBARI-24913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated AMBARI-24913:
Labels: pull-request-available (was: )
> Update ldap-configuration category in Ambari Configurations data to contain
> properties to aid in automated LDAP configuration
> -
>
> Key: AMBARI-24913
> URL: https://issues.apache.org/jira/browse/AMBARI-24913
> Project: Ambari
> Issue Type: Task
> Components: ambari-server, security
>Affects Versions: 2.7.0
>Reporter: Sandor Molnar
>Assignee: Sandor Molnar
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
> Update {{ldap-configuration}} category in Ambari Configurations data to
> contain properties to aid in automated LDAP configuration:
> * {{ambari.ldap.manage_services}}
> ** This property is used to indicate whether Ambari is to manage relevant
> services' LDAP configurations or not ("true" | "false")
> * {{ambari.ldap.enabled_services}}
> ** This property is used to declare what services are expected to be
> configured for LDAP and is expected to be a comma-delimited list of services
> or "{{*}}" to indicate all services.
> Examples:
> {code:java|title=All services}
> "ambari.ldap.manage_services":"true",
> "ambari.ldap.enabled_services":"*"
> {code}
> {code:java|title=Only Ranger}
> "ambari.ldap.manage_services":"true",
> "ambari.ldap.enabled_services":"Ranger"
> {code}
> {code:java|title=Ranger, and Atlas}
> "ambari.ldap.manage_services":"true",
> "ambari.ldap.enabled_services":"Ranger, Atlas"
> {code}
> {code:java|title=Do not manage services}
> "ambari.ldap.manage_services":"false",
> "ambari.ldap.enabled_services":""
> {code}
> Each service in the set of services should have indicated it supports LDAP
> (see BUG-114409) else it will silently be ignored.
> This value should be set via Ambari's REST API or a Blueprint.
> Upon setting this value via the Ambari REST API, it is expected that internal
> logic will be triggered to ensure the relevant services in the list are
> configured for LDAP or not configured for LDAP as the case may be.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24920) LocalFS (file:///) directory creation fails
[ https://issues.apache.org/jira/browse/AMBARI-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691610#comment-16691610 ] Hudson commented on AMBARI-24920: - SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #10306 (See [https://builds.apache.org/job/Ambari-trunk-Commit/10306/]) AMBARI-24920. LocalFS (file:///) directory creation fails (aonishuk) (aonishuk: [https://gitbox.apache.org/repos/asf?p=ambari.git&a=commit&h=3da344ab8e7d878868998ee514dd0dbc99a2be55]) * (edit) ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py * (add) ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py.rej > LocalFS (file:///) directory creation fails > --- > > Key: AMBARI-24920 > URL: https://issues.apache.org/jira/browse/AMBARI-24920 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Attachments: AMBARI-24920.patch, AMBARI-24920.patch > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24904) JAR does not exist: /var/lib/ambari-agent/lib/fast-hdfs-resource.jar
[ https://issues.apache.org/jira/browse/AMBARI-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691611#comment-16691611 ] Hudson commented on AMBARI-24904: - SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #10306 (See [https://builds.apache.org/job/Ambari-trunk-Commit/10306/]) AMBARI-24904. JAR does not exist: (aonishuk: [https://gitbox.apache.org/repos/asf?p=ambari.git&a=commit&h=9e6965d0bff7d31fc188d9b5be42aac80f94e286]) * (edit) ambari-common/src/main/python/resource_management/libraries/providers/hdfs_resource.py > JAR does not exist: /var/lib/ambari-agent/lib/fast-hdfs-resource.jar > > > Key: AMBARI-24904 > URL: https://issues.apache.org/jira/browse/AMBARI-24904 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Attachments: AMBARI-24904.patch, AMBARI-24904.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24921) If service does not have client service component created then "Run Service Check" option should be made hidden
[ https://issues.apache.org/jira/browse/AMBARI-24921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AMBARI-24921: Labels: pull-request-available (was: ) > If service does not have client service component created then "Run Service > Check" option should be made hidden > --- > > Key: AMBARI-24921 > URL: https://issues.apache.org/jira/browse/AMBARI-24921 > Project: Ambari > Issue Type: Task > Components: ambari-web >Affects Versions: 2.7.0 >Reporter: Andrii Tkach >Assignee: Andrii Tkach >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > > If service does not have client service component created then "Run Service > Check" option should be made hidden -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24921) If service does not have client service component created then "Run Service Check" option should be made hidden
[ https://issues.apache.org/jira/browse/AMBARI-24921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrii Tkach updated AMBARI-24921: -- Description: If service does not have client service component created then "Run Service Check" option should be made hidden > If service does not have client service component created then "Run Service > Check" option should be made hidden > --- > > Key: AMBARI-24921 > URL: https://issues.apache.org/jira/browse/AMBARI-24921 > Project: Ambari > Issue Type: Task > Components: ambari-web >Affects Versions: 2.7.0 >Reporter: Andrii Tkach >Assignee: Andrii Tkach >Priority: Major > Fix For: 2.8.0 > > > If service does not have client service component created then "Run Service > Check" option should be made hidden -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMBARI-24921) If service does not have client service component created then "Run Service Check" option should be made hidden
Andrii Tkach created AMBARI-24921: - Summary: If service does not have client service component created then "Run Service Check" option should be made hidden Key: AMBARI-24921 URL: https://issues.apache.org/jira/browse/AMBARI-24921 Project: Ambari Issue Type: Task Components: ambari-web Affects Versions: 2.7.0 Reporter: Andrii Tkach Assignee: Andrii Tkach Fix For: 2.8.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24904) JAR does not exist: /var/lib/ambari-agent/lib/fast-hdfs-resource.jar
[ https://issues.apache.org/jira/browse/AMBARI-24904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-24904: - Attachment: AMBARI-24904.patch > JAR does not exist: /var/lib/ambari-agent/lib/fast-hdfs-resource.jar > > > Key: AMBARI-24904 > URL: https://issues.apache.org/jira/browse/AMBARI-24904 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Attachments: AMBARI-24904.patch, AMBARI-24904.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24920) LocalFS (file:///) directory creation fails
[ https://issues.apache.org/jira/browse/AMBARI-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AMBARI-24920: Labels: pull-request-available (was: ) > LocalFS (file:///) directory creation fails > --- > > Key: AMBARI-24920 > URL: https://issues.apache.org/jira/browse/AMBARI-24920 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Attachments: AMBARI-24920.patch, AMBARI-24920.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24905) Service display name on left navigation bar should be suffixed with "Client" if only client service component is present for a service
[ https://issues.apache.org/jira/browse/AMBARI-24905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691516#comment-16691516 ] Hudson commented on AMBARI-24905: - SUCCESS: Integrated in Jenkins build Ambari-trunk-Commit #10305 (See [https://builds.apache.org/job/Ambari-trunk-Commit/10305/]) AMBARI-24905 Service display name on left navigation bar should be (1963907+atkach: [https://gitbox.apache.org/repos/asf?p=ambari.git&a=commit&h=72213a6452550921485e26f2e64230ad26cf8a16]) * (edit) ambari-web/app/models/service.js * (edit) ambari-web/app/templates/main/service/menu_item.hbs > Service display name on left navigation bar should be suffixed with "Client" > if only client service component is present for a service > -- > > Key: AMBARI-24905 > URL: https://issues.apache.org/jira/browse/AMBARI-24905 > Project: Ambari > Issue Type: Task > Components: ambari-web >Affects Versions: 2.7.0 >Reporter: Andrii Tkach >Assignee: Andrii Tkach >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > Attachments: Displayname HDFS Client.png, Dsplayname HDFS.png > > Time Spent: 1h > Remaining Estimate: 0h > > When a service only contains client service component then check service's > displayname. If service's displayname does not and with "Client" then suffix > service's displayname with "Client" > Current behavior: !Dsplayname HDFS.png! > Expected behavior: !Displayname HDFS Client.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24920) LocalFS (file:///) directory creation fails
[ https://issues.apache.org/jira/browse/AMBARI-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-24920: - Attachment: AMBARI-24920.patch > LocalFS (file:///) directory creation fails > --- > > Key: AMBARI-24920 > URL: https://issues.apache.org/jira/browse/AMBARI-24920 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Fix For: 2.8.0 > > Attachments: AMBARI-24920.patch, AMBARI-24920.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AMBARI-24920) LocalFS (file:///) directory creation fails
Andrew Onischuk created AMBARI-24920: Summary: LocalFS (file:///) directory creation fails Key: AMBARI-24920 URL: https://issues.apache.org/jira/browse/AMBARI-24920 Project: Ambari Issue Type: Bug Reporter: Andrew Onischuk Assignee: Andrew Onischuk Fix For: 2.8.0 Attachments: AMBARI-24920.patch -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24920) LocalFS (file:///) directory creation fails
[ https://issues.apache.org/jira/browse/AMBARI-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-24920: - Status: Patch Available (was: Open) > LocalFS (file:///) directory creation fails > --- > > Key: AMBARI-24920 > URL: https://issues.apache.org/jira/browse/AMBARI-24920 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Fix For: 2.8.0 > > Attachments: AMBARI-24920.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24920) LocalFS (file:///) directory creation fails
[ https://issues.apache.org/jira/browse/AMBARI-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Onischuk updated AMBARI-24920: - Attachment: AMBARI-24920.patch > LocalFS (file:///) directory creation fails > --- > > Key: AMBARI-24920 > URL: https://issues.apache.org/jira/browse/AMBARI-24920 > Project: Ambari > Issue Type: Bug >Reporter: Andrew Onischuk >Assignee: Andrew Onischuk >Priority: Major > Fix For: 2.8.0 > > Attachments: AMBARI-24920.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691477#comment-16691477 ] Hudson commented on AMBARI-24833: - SUCCESS: Integrated in Jenkins build Ambari-LogSearch-master-Commit #35 (See [https://builds.apache.org/job/Ambari-LogSearch-master-Commit/35/]) AMBARI-24833. Support for cloud logs to using filters + JSON output (github: [https://gitbox.apache.org/repos/asf?p=ambari-logsearch.git&a=commit&h=72b3dd6d39d233db5b02536b3c6a3215971f6cd6]) * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/CloudStorageInputConfigHandler.java * (edit) ambari-logsearch-logfeeder/src/main/resources/logfeeder.properties * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageLoggerFactory.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageUploader.java * (edit) ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/AbstractInputConfigHandler.java * (edit) ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java * (add) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputLineEnricher.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/conf/LogFeederProps.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/DefaultInputConfigHandler.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputManagerImpl.java * (edit) ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/cloud/CloudStorageOutputManager.java > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691475#comment-16691475 ] ASF GitHub Bot commented on AMBARI-24833: - asfgit commented on issue #27: AMBARI-24833. HDFS client kerberos support + small fixes URL: https://github.com/apache/ambari-logsearch/pull/27#issuecomment-439831367 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/Ambari-LogSearch-Github-PR-Builder/57/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMBARI-24919) Allow Namenode HA setup with external namenodes in BlueprintConfigurationProcessor
[ https://issues.apache.org/jira/browse/AMBARI-24919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated AMBARI-24919: Labels: pull-request-available (was: ) > Allow Namenode HA setup with external namenodes in > BlueprintConfigurationProcessor > -- > > Key: AMBARI-24919 > URL: https://issues.apache.org/jira/browse/AMBARI-24919 > Project: Ambari > Issue Type: Task >Affects Versions: 2.7.2 >Reporter: Balázs Bence Sári >Assignee: Balázs Bence Sári >Priority: Critical > Labels: pull-request-available > Fix For: trunk > > > There is an increasing demand for clusters with some of the components being > managed externally to Ambari. > This ticket addresses the problem where the cluster contains a HDFS_CLIENT > but server HDFS server components are external and NAMENODE is HA. > In this cases, cluster template validation failed as there were 0 name nodes > which is less than the expected minimum of two. > Going forward the following setup should pass validation: > - Namenode HA is enabled > - There are 0 namenodes > - all namenode dfs rpc addresses point to an external fqdn. > This will give sufficent protection against accidentally omitted namenodes, > however will enable clusters with external namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[ https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691466#comment-16691466 ] ASF GitHub Bot commented on AMBARI-24833: - oleewere opened a new pull request #27: AMBARI-24833. HDFS client kerberos support + small fixes URL: https://github.com/apache/ambari-logsearch/pull/27 # What changes were proposed in this pull request? - add kerberos support for hdfs client - remove external hdfs client -> merge it with hdfs client - small fixes ## How was this patch tested? FT: manually - docker/vagrant/gce env This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Log Feeder: send logs to cloud storage (gcs/s3 etc.) > > > Key: AMBARI-24833 > URL: https://issues.apache.org/jira/browse/AMBARI-24833 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.7.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó >Priority: Major > Labels: pull-request-available > Fix For: 2.8.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691460#comment-16691460
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere closed pull request #26: AMBARI-24833. Support for cloud logs to using
filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
b/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
index f6cb519b95..12af637d8c 100644
---
a/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
+++
b/ambari-logsearch-config-local/src/main/java/org/apache/ambari/logsearch/config/local/LogSearchConfigLogFeederLocal.java
@@ -84,12 +84,7 @@ public void monitorInputConfigChanges(final
InputConfigMonitor inputConfigMonito
File[] inputConfigFiles = new
File(configDir).listFiles(inputConfigFileFilter);
if (inputConfigFiles != null) {
for (File inputConfigFile : inputConfigFiles) {
-String inputConfig = new
String(Files.readAllBytes(inputConfigFile.toPath()));
-Matcher m = serviceNamePattern.matcher(inputConfigFile.getName());
-m.find();
-String serviceName = m.group(1);
-JsonElement inputConfigJson =
JsonHelper.mergeGlobalConfigWithInputConfig(parser, inputConfig,
globalConfigNode);
-inputConfigMonitor.loadInputConfigs(serviceName,
InputConfigGson.gson.fromJson(inputConfigJson, InputConfigImpl.class));
+tryLoadingInputConfig(inputConfigMonitor, parser, globalConfigNode,
inputConfigFile);
}
}
final FileSystem fs = FileSystems.getDefault();
@@ -100,6 +95,41 @@ public void monitorInputConfigChanges(final
InputConfigMonitor inputConfigMonito
executorService.submit(updater);
}
+ private void tryLoadingInputConfig(InputConfigMonitor inputConfigMonitor,
JsonParser parser, JsonArray globalConfigNode, File inputConfigFile) throws
Exception {
+// note: that will try to solve a intermittent issue when the input config
json is a null string (during file generation), that process will re-try to
process the files a few times
+int tries = 0;
+while(true) {
+ tries++;
+ Matcher m = serviceNamePattern.matcher(inputConfigFile.getName());
+ m.find();
+ String inputConfig = new
String(Files.readAllBytes(inputConfigFile.toPath()));
+ String serviceName = m.group(1);
+ JsonElement inputConfigJson = null;
+ logger.info("Trying to load '{}' service input config from input file
'{}'", serviceName, inputConfigFile.getAbsolutePath());
+ try {
+inputConfigJson = JsonHelper.mergeGlobalConfigWithInputConfig(parser,
inputConfig, globalConfigNode);
+ } catch (Exception e) {
+final String errorMessage;
+if (tries < 3) {
+ errorMessage = String.format("Cannot parse input config: %s, will
retry in a few seconds again (tries: %s)", inputConfig, String.valueOf(tries));
+ logger.error(errorMessage, e);
+ try {
+Thread.sleep(2000);
+ } catch (Exception ex) {
+// skip
+ }
+ continue;
+} else {
+ errorMessage = String.format("Cannot parse input config: %s, after
%s tries. Will skip to processing it", inputConfig, String.valueOf(tries));
+ logger.error(errorMessage, e);
+ break;
+}
+ }
+ inputConfigMonitor.loadInputConfigs(serviceName,
InputConfigGson.gson.fromJson(inputConfigJson, InputConfigImpl.class));
+ break;
+}
+ }
+
@Override
public void close() throws IOException {
}
diff --git
a/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java
b/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java
index fd08e07fed..0975c395fa 100644
---
a/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java
+++
b/ambari-logsearch-config-zookeeper/src/main/java/org/apache/ambari/logsearch/config/zookeeper/LogLevelFilterManagerZK.java
@@ -48,6 +48,7 @@
public LogLevelFilterManagerZK(Map properties) throws
Exception {
this.client = LogSearchConfigZKHelper.createZKClient(properties);
+this.client.start();
this.serverCache = new TreeCache(client, "/");
this.aclList = LogSearchConfigZKHelper.getA
[jira] [Created] (AMBARI-24919) Allow Namenode HA setup with external namenodes in BlueprintConfigurationProcessor
Balázs Bence Sári created AMBARI-24919: -- Summary: Allow Namenode HA setup with external namenodes in BlueprintConfigurationProcessor Key: AMBARI-24919 URL: https://issues.apache.org/jira/browse/AMBARI-24919 Project: Ambari Issue Type: Task Affects Versions: 2.7.2 Reporter: Balázs Bence Sári Assignee: Balázs Bence Sári Fix For: trunk There is an increasing demand for clusters with some of the components being managed externally to Ambari. This ticket addresses the problem where the cluster contains a HDFS_CLIENT but server HDFS server components are external and NAMENODE is HA. In this cases, cluster template validation failed as there were 0 name nodes which is less than the expected minimum of two. Going forward the following setup should pass validation: - Namenode HA is enabled - There are 0 namenodes - all namenode dfs rpc addresses point to an external fqdn. This will give sufficent protection against accidentally omitted namenodes, however will enable clusters with external namenodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691454#comment-16691454
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #26: AMBARI-24833. Support for
cloud logs to using filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26#discussion_r234544396
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputLineEnricher.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output;
+
+import com.google.common.hash.Hashing;
+import org.apache.ambari.logfeeder.plugin.common.MetricData;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.ambari.logfeeder.plugin.input.InputMarker;
+import org.apache.ambari.logfeeder.util.LogFeederUtil;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Utility class for fill output with other fields
+ */
+public class OutputLineEnricher {
+
+ private static final Logger logger =
LogManager.getLogger(OutputLineEnricher.class);
+
+ private static final int MAX_OUTPUT_SIZE = 32765; // 32766-1
+
+ public void enrichFields(final Map jsonObj, final
InputMarker inputMarker, final MetricData messageTruncateMetric) {
+Input input = inputMarker.getInput();
+// Update the block with the context fields
+for (Map.Entry entry :
input.getInputDescriptor().getAddFields().entrySet()) {
+ if (jsonObj.get(entry.getKey()) == null ||
entry.getKey().equals("cluster") && "null".equals(jsonObj.get(entry.getKey(
{
Review comment:
i will solve that in a next commit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691455#comment-16691455
]
ASF GitHub Bot commented on AMBARI-24833:
-
oleewere commented on a change in pull request #26: AMBARI-24833. Support for
cloud logs to using filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26#discussion_r23455
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/AbstractInputConfigHandler.java
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.manager.operations.impl;
+
+import org.apache.ambari.logfeeder.manager.InputConfigHolder;
+import org.apache.ambari.logfeeder.manager.operations.InputConfigHandler;
+import org.apache.ambari.logfeeder.plugin.common.AliasUtil;
+import org.apache.ambari.logfeeder.plugin.filter.Filter;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.FilterDescriptor;
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Holds common operations for input config handlers
+ */
+public abstract class AbstractInputConfigHandler implements InputConfigHandler
{
+
+ private static final Logger logger =
LogManager.getLogger(AbstractInputConfigHandler.class);
+
+ protected void loadFilters(String serviceName, InputConfigHolder
inputConfigHolder) {
+sortFilters(inputConfigHolder);
+
+List toRemoveInputList = new ArrayList<>();
+for (Input input :
inputConfigHolder.getInputManager().getInputList(serviceName)) {
+ for (FilterDescriptor filterDescriptor :
inputConfigHolder.getFilterConfigList()) {
+if (filterDescriptor == null) {
+ logger.warn("Filter descriptor is smpty. Skipping...");
Review comment:
i will solve that in a next commit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AMBARI-24918) Infra Manager: ssl support
Krisztian Kasa created AMBARI-24918: --- Summary: Infra Manager: ssl support Key: AMBARI-24918 URL: https://issues.apache.org/jira/browse/AMBARI-24918 Project: Ambari Issue Type: Improvement Components: ambari-infra Affects Versions: 2.8.0 Reporter: Krisztian Kasa Assignee: Krisztian Kasa Fix For: 2.8.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691379#comment-16691379
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #26: AMBARI-24833. Support for
cloud logs to using filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26#discussion_r234520580
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/manager/operations/impl/AbstractInputConfigHandler.java
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.manager.operations.impl;
+
+import org.apache.ambari.logfeeder.manager.InputConfigHolder;
+import org.apache.ambari.logfeeder.manager.operations.InputConfigHandler;
+import org.apache.ambari.logfeeder.plugin.common.AliasUtil;
+import org.apache.ambari.logfeeder.plugin.filter.Filter;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import
org.apache.ambari.logsearch.config.api.model.inputconfig.FilterDescriptor;
+import org.apache.commons.lang.BooleanUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Holds common operations for input config handlers
+ */
+public abstract class AbstractInputConfigHandler implements InputConfigHandler
{
+
+ private static final Logger logger =
LogManager.getLogger(AbstractInputConfigHandler.class);
+
+ protected void loadFilters(String serviceName, InputConfigHolder
inputConfigHolder) {
+sortFilters(inputConfigHolder);
+
+List toRemoveInputList = new ArrayList<>();
+for (Input input :
inputConfigHolder.getInputManager().getInputList(serviceName)) {
+ for (FilterDescriptor filterDescriptor :
inputConfigHolder.getFilterConfigList()) {
+if (filterDescriptor == null) {
+ logger.warn("Filter descriptor is smpty. Skipping...");
Review comment:
typo: smpty -> empty
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMBARI-24833) Log Feeder: send logs to cloud storage (gcs/s3 etc.)
[
https://issues.apache.org/jira/browse/AMBARI-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691380#comment-16691380
]
ASF GitHub Bot commented on AMBARI-24833:
-
kasakrisz commented on a change in pull request #26: AMBARI-24833. Support for
cloud logs to using filters + JSON output
URL: https://github.com/apache/ambari-logsearch/pull/26#discussion_r234521392
##
File path:
ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/output/OutputLineEnricher.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.ambari.logfeeder.output;
+
+import com.google.common.hash.Hashing;
+import org.apache.ambari.logfeeder.plugin.common.MetricData;
+import org.apache.ambari.logfeeder.plugin.input.Input;
+import org.apache.ambari.logfeeder.plugin.input.InputMarker;
+import org.apache.ambari.logfeeder.util.LogFeederUtil;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Utility class for fill output with other fields
+ */
+public class OutputLineEnricher {
+
+ private static final Logger logger =
LogManager.getLogger(OutputLineEnricher.class);
+
+ private static final int MAX_OUTPUT_SIZE = 32765; // 32766-1
+
+ public void enrichFields(final Map jsonObj, final
InputMarker inputMarker, final MetricData messageTruncateMetric) {
+Input input = inputMarker.getInput();
+// Update the block with the context fields
+for (Map.Entry entry :
input.getInputDescriptor().getAddFields().entrySet()) {
+ if (jsonObj.get(entry.getKey()) == null ||
entry.getKey().equals("cluster") && "null".equals(jsonObj.get(entry.getKey(
{
Review comment:
```"cluster".equals(entry.getKey())```
would be safer
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Log Feeder: send logs to cloud storage (gcs/s3 etc.)
>
>
> Key: AMBARI-24833
> URL: https://issues.apache.org/jira/browse/AMBARI-24833
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.7.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.8.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
