On November 21, 2017 5:48 pm, Clark Boylan wrote:
On Tue, Nov 21, 2017, at 09:17 AM, Tristan Cacqueray wrote:snipActually the rfc is this thread :-) Though I forgot to mention the first steps that could use comments before we move on: * create the openstack-infra/log-classify project, * import the log-classify.crm script, * wrap the script with a more user friendly interface, and * modify the puppet-log_processor to use that new project insteadThis sounds like a great place to start. Considering the interest already forming around this I would say go ahead and create the project and start with the import process so that people have a concrete place to start working on this. I am sure it will evolve from there, but getting started is often the most difficult step. Related to the last step we have temporarily disabled CRM classification in the log processor pipeline because we treat the whole file path as a unique file to classify which ended up filling our workers' disks with classification files. I think one of the things we will want to address early on is using the basename rather than the whole path to significantly reduce the total number of data files on disk. This way we can get it running in the log processor pipeline again for proper production feedback of changes that are happening. Once again let me know if I can help with anything (happy to review new project creation changes for example).
Excellent, project creation is proposed here: https://review.openstack.org/#/q/topic:log-classify I'm open to suggestion regarding the name and structure of the project. Otherwise I'll create a standard openstack python project with: logclassify.logstash module to interface with the script using the design of the log-gearman-client.py (e.g. a process(event)). logclassify.cmd module to use the script standalone. And then write a first test and implementation of that basename base data files improvement. If that works ok, then a follow-up change will modify the log-gearman-client to import logclassify instead of running the script directly.
Thank you for getting this started, Clark
Thanks for the quick feedback! -Tristan
pgpQdQSYGXoTo.pgp
Description: PGP signature
_______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
