Re: Review Request 53663: Add ability to add custom input descriptor to logfeeder

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53663/
---

(Updated Nov. 15, 2016, 10:22 p.m.)


Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.


Changes
---

updated to be appliable on the latest branch-2.5


Bugs: AMBARI-18854
https://issues.apache.org/jira/browse/AMBARI-18854


Repository: ambari


Description
---

Offer some properties in the logsearch configuration via which the user can set 
up some other input logs in a similar way as the component logs.


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_config_aggregator.py
 ad515ee 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
 1f5064a 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
 e90d623 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/HadoopServiceConfig.json.j2
 7e9153f 
  ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logfeeder.py 7a13637 

Diff: https://reviews.apache.org/r/53663/diff/


Testing
---

Tested on local cluster.


Thanks,

Miklos Gergely

Re: Review Request 53663: Add ability to add custom input descriptor to logfeeder

[Robert Nettleton]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53663/#review155816
---


Ship it!




Ship It!

- Robert Nettleton


On Nov. 14, 2016, 12:32 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53663/
> ---
> 
> (Updated Nov. 14, 2016, 12:32 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18854
> https://issues.apache.org/jira/browse/AMBARI-18854
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Offer some properties in the logsearch configuration via which the user can 
> set up some other input logs in a similar way as the component logs.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_config_aggregator.py
>  2e9dda3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  1f5064a 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
>  e90d623 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/HadoopServiceConfig.json.j2
>  7e9153f 
>   ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logfeeder.py 
> 7a13637 
> 
> Diff: https://reviews.apache.org/r/53663/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster.
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 53663: Add ability to add custom input descriptor to logfeeder

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53663/#review155807
---


Ship it!




Ship It!

- Oliver Szabo


On Nov. 14, 2016, 12:32 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53663/
> ---
> 
> (Updated Nov. 14, 2016, 12:32 p.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18854
> https://issues.apache.org/jira/browse/AMBARI-18854
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Offer some properties in the logsearch configuration via which the user can 
> set up some other input logs in a similar way as the component logs.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_config_aggregator.py
>  2e9dda3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  1f5064a 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
>  e90d623 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/HadoopServiceConfig.json.j2
>  7e9153f 
>   ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logfeeder.py 
> 7a13637 
> 
> Diff: https://reviews.apache.org/r/53663/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster.
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 53663: Add ability to add custom input descriptor to logfeeder

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53663/
---

(Updated Nov. 14, 2016, 12:32 p.m.)


Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.


Changes
---

Support multiple custom services


Bugs: AMBARI-18854
https://issues.apache.org/jira/browse/AMBARI-18854


Repository: ambari


Description
---

Offer some properties in the logsearch configuration via which the user can set 
up some other input logs in a similar way as the component logs.


Diffs (updated)
-

  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_config_aggregator.py
 2e9dda3 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
 1f5064a 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
 e90d623 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/HadoopServiceConfig.json.j2
 7e9153f 
  ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logfeeder.py 7a13637 

Diff: https://reviews.apache.org/r/53663/diff/


Testing
---

Tested on local cluster.


Thanks,

Miklos Gergely

Re: Review Request 53663: Add ability to add custom input descriptor to logfeeder

[Oliver Szabo]

> On Nov. 11, 2016, 1:05 a.m., Oliver Szabo wrote:
> > ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml,
> >  line 22
> > 
> >
> > what happens if i want to add multiple custom labels/components to 
> > logsearch?
> 
> Miklos Gergely wrote:
> This solution supports only one label, which may have as many 
> components/logs as we wish for. Do we want to allow the user to add multiple 
> labels tis way? That would be a bit tricky, but certainly possible.

yes, that should be flexible, we should not force to user the have only one 
label. maybe you can find different way to do it if its easier (not as the same 
how we do in the conifg agregator...e.g.: extend the json content through 
configs for HadoopServicesConfig.json)


> On Nov. 11, 2016, 1:05 a.m., Oliver Szabo wrote:
> > ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py,
> >  line 150
> > 
> >
> > why add this is needed in this patch? 
> > we should remove these with hst anyway after nifi/hst is updated. if 
> > ambari 2.4 is used its there, for ambari 2.5 that wont be here
> 
> Miklos Gergely wrote:
> Currently input.config-nifi.json.j2 is still present in the ambari-server 
> project, so this should be in the params.py as well in order to be 
> consistent. Or we may as well remove both, would that be OK?

you can keep this as is, but we will remove it anyway


- Oliver


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53663/#review155679
---


On Nov. 11, 2016, 12:39 a.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53663/
> ---
> 
> (Updated Nov. 11, 2016, 12:39 a.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18854
> https://issues.apache.org/jira/browse/AMBARI-18854
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Offer some properties in the logsearch configuration via which the user can 
> set up some other input logs in a similar way as the component logs.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_config_aggregator.py
>  2e9dda3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  1f5064a 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
>  e90d623 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/HadoopServiceConfig.json.j2
>  7e9153f 
>   ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logfeeder.py 
> 7a13637 
> 
> Diff: https://reviews.apache.org/r/53663/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster.
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 53663: Add ability to add custom input descriptor to logfeeder

[Miklos Gergely]

> On Nov. 11, 2016, 1:05 a.m., Oliver Szabo wrote:
> > ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py,
> >  line 150
> > 
> >
> > why add this is needed in this patch? 
> > we should remove these with hst anyway after nifi/hst is updated. if 
> > ambari 2.4 is used its there, for ambari 2.5 that wont be here

Currently input.config-nifi.json.j2 is still present in the ambari-server 
project, so this should be in the params.py as well in order to be consistent. 
Or we may as well remove both, would that be OK?


> On Nov. 11, 2016, 1:05 a.m., Oliver Szabo wrote:
> > ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py,
> >  line 70
> > 
> >
> > why is that file deleted?

Because if the user enters some custom input config first, and later deletes it 
then the generated custom input file should disappear too. To ensure this the 
file should be deleted every time, and potentially re-generated if there is 
some custom configured input.


> On Nov. 11, 2016, 1:05 a.m., Oliver Szabo wrote:
> > ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml,
> >  line 22
> > 
> >
> > what happens if i want to add multiple custom labels/components to 
> > logsearch?

This solution supports only one label, which may have as many components/logs 
as we wish for. Do we want to allow the user to add multiple labels tis way? 
That would be a bit tricky, but certainly possible.


- Miklos


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53663/#review155679
---


On Nov. 11, 2016, 12:39 a.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53663/
> ---
> 
> (Updated Nov. 11, 2016, 12:39 a.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18854
> https://issues.apache.org/jira/browse/AMBARI-18854
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Offer some properties in the logsearch configuration via which the user can 
> set up some other input logs in a similar way as the component logs.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_config_aggregator.py
>  2e9dda3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  1f5064a 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
>  e90d623 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/HadoopServiceConfig.json.j2
>  7e9153f 
>   ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logfeeder.py 
> 7a13637 
> 
> Diff: https://reviews.apache.org/r/53663/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster.
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Re: Review Request 53663: Add ability to add custom input descriptor to logfeeder

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53663/#review155679
---




ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml
 (line 22)


what happens if i want to add multiple custom labels/components to 
logsearch?



ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
 (line 150)


why add this is needed in this patch? 
we should remove these with hst anyway after nifi/hst is updated. if ambari 
2.4 is used its there, for ambari 2.5 that wont be here



ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
 (line 70)


why is that file deleted?


- Oliver Szabo


On Nov. 11, 2016, 12:39 a.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53663/
> ---
> 
> (Updated Nov. 11, 2016, 12:39 a.m.)
> 
> 
> Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-18854
> https://issues.apache.org/jira/browse/AMBARI-18854
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Offer some properties in the logsearch configuration via which the user can 
> set up some other input logs in a similar way as the component logs.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_config_aggregator.py
>  2e9dda3 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
>  1f5064a 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
>  e90d623 
>   
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/HadoopServiceConfig.json.j2
>  7e9153f 
>   ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logfeeder.py 
> 7a13637 
> 
> Diff: https://reviews.apache.org/r/53663/diff/
> 
> 
> Testing
> ---
> 
> Tested on local cluster.
> 
> 
> Thanks,
> 
> Miklos Gergely
> 
>

Review Request 53663: Add ability to add custom input descriptor to logfeeder

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53663/
---

Review request for Ambari, Oliver Szabo, Robert Nettleton, and Sumit Mohanty.


Bugs: AMBARI-18854
https://issues.apache.org/jira/browse/AMBARI-18854


Repository: ambari


Description
---

Offer some properties in the logsearch configuration via which the user can set 
up some other input logs in a similar way as the component logs.


Diffs
-

  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/configuration/logfeeder-custom-logsearch-conf.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/logsearch_config_aggregator.py
 2e9dda3 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
 1f5064a 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logfeeder.py
 e90d623 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/templates/HadoopServiceConfig.json.j2
 7e9153f 
  ambari-server/src/test/python/stacks/2.4/LOGSEARCH/test_logfeeder.py 7a13637 

Diff: https://reviews.apache.org/r/53663/diff/


Testing
---

Tested on local cluster.


Thanks,

Miklos Gergely