I will continue working on my implementation of JEP-210 for pipelines the API is simple to implement and extend https://github.com/kuisathaverat/pipeline-filebeat-logs so far it works fine on a production-grade instance, there are a few details I want to change. For the JEP-207 I think I will not try to use something similar it seems too complex to simply send the logs to external storage, I will simply redirect the logs and that's it, we do not have too many FreeStyle jobs and we can convert them to pipelines.
El miércoles, 19 de mayo de 2021 a las 19:12:46 UTC+2, Oleg Nenashev escribió: > > The logstash plugin can forward logs to for both pipeline and freestyle > to elastic search and others. But it doesn't support to read the logs from > there to display in Jenkins, what it forwards is rather heavy weight and > for pipeline jobs not the complete output is transferred (all the output > from the pipeline core not coming from a step itself is missing, like the > first lines with what caused the build and the final line with the result) > > I had a PoC for pluggable log storage and the Logstash plugin in 2016... > We started this project but then I had to stop working on that due to the > reasons outside my control. FTR > https://speakerdeck.com/onenashev/jw2016-hackathon-external-build-logging-prototype-demo > > FWIW it is much easier to implement a push-only pluggable log storage than > one with visualization. In JEP-207 / JEP-210 Jesse and me designed a > seamless integration with runtime visualization support. It ended up being > a massive time overhead compared to be a push-only solution. > > > On Saturday, May 15, 2021 at 2:12:42 PM UTC+2 Markus Winter wrote: > >> The logstash plugin can forward logs to for both pipeline and freestyle >> to elastic search and others. >> But it doesn't support to read the logs from there to display in Jenkins, >> what it forwards is rather heavy weight and for pipeline jobs not the >> complete output is transferred (all the output from the pipeline core not >> coming from a step itself is missing, like the first lines with what caused >> the build and the final line with the result) >> >> There is an implementation for what you want to do at >> https://github.com/SAP/elasticsearch-logs-plugin (inspired by the >> cloudwatch logs plugin) >> >> This plugin was mainly implemented to forward the logs of a >> JenkinsFileRunner from inside a short living Kubernetes pod so it is >> available outside. It is partially generic as it has an extension point >> (similar to logstash plugin) and we implemented also forwarding to fluentd >> (that will then forward to Elastic Search, this has proven to be more >> robust in our scenario). >> It can also read from ES, but performance is rather bad here. Also some >> plugins still rely on the log being available in the file system, so we >> decided to have both though this meant to copy code from the pipeline >> plugins. >> It can also forward logs of Freestyle jobs. >> >> The plugin also provides eventing for pipeline steps and transfers much >> less data than the logstash plugin. >> >> >> >> Am 10.05.2021 um 18:43 schrieb Oleg Nenashev: >> >> Yes, JEP-207 was abandoned after, ehm some reorg at my company. Great to >> see my former teammates commenting in this thread :D >> >> JEP-207 was indeed abandoned, but there were pull requests submitted to >> the Jenkins core and reference implementations (see the JEP for links). We >> had some disagreements with Jesse on APIs, especially w.r.t splitting the >> Log Storage and the Log Browser extensions (I needed it for Logstash/ELK >> where there might be multiple browsing implementations). But overall the >> API work was ready for merge with @Beta annotations IMHO. Same goes to >> External Log Storage API plugin. Note that JEP-207 was designed not only >> for AbstractProject-alike jobs, but it also added foundations for external >> Task Logs which is also a good use-case for centralized log systems. >> >> The work could be definitely finished. As Jesse said, the most >> complicated part would be ensuring JEP-207 and JEP-210 interoperability >> without performance loss. Also there would be a major merge conflict if we >> recover the old PR I'd guess. >> >> If somebody is interested, I am interested to participate as much as my >> time allows. Spoiler alert: not so much >> >> >> >> >> >> On Monday, May 10, 2021 at 5:36:34 PM UTC+2 Jesse Glick wrote: >> >>> On Mon, May 10, 2021 at 11:11 AM kuisathaverat <kuisat...@gmail.com> >>> wrote: >>> >>>> it [is] only for pipelines so I have taken a look at JEP-207: External >>>> Build Logging support in the Jenkins Core >>>> <https://github.com/jenkinsci/jep/blob/master/jep/207/README.adoc> >>>> but it seems abandoned >>>> >>> >>> Yes. Something like that could be resurrected if there is a need, with >>> https://javadoc.jenkins.io/plugin/workflow-api/org/jenkinsci/plugins/workflow/log/package-summary.html >>> >>> reworked to use the core API (there are still some Pipeline-specific >>> aspects such as step delineation). >>> >>> I am personally less interested in supporting non-Pipeline builds as in >>> addressing the key architectural limitation of current JEP-210 >>> implementations, which is that Stapler and thus core classes involved in >>> log handling use `long` as a cursor rather than an opaque token, making it >>> impossible to offer high-performance log rendering inside Jenkins >>> (“classic”, Blue Ocean, etc.) particularly for running builds. >>> >>> what is the best approach to implement a distributed log system for >>>> FreeStyle jobs (and others), a ConsoleLogFilter? >>>> >>> >>> Several longstanding plugins are able to *mirror* (non-Pipeline) log >>> output to external systems. Without something like JEP-207 you cannot have >>> the external system be the source of truth: Jenkins will continue streaming >>> the log to a local file, and reading it from there. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to jenkinsci-de...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-dev/453bf8e6-30b9-43ce-90c2-b4f4dc7d7f78n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/jenkinsci-dev/453bf8e6-30b9-43ce-90c2-b4f4dc7d7f78n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6cbaa684-9d4e-4960-8288-d7d967a4eabbn%40googlegroups.com.
