The logstash plugin can forward logs to for both pipeline and freestyle
to elastic search and others.
But it doesn't support to read the logs from there to display in
Jenkins, what it forwards is rather heavy weight and for pipeline jobs
not the complete output is transferred (all the output from the pipeline
core not coming from a step itself is missing, like the first lines with
what caused the build and the final line with the result)
There is an implementation for what you want to do at
https://github.com/SAP/elasticsearch-logs-plugin (inspired by the
cloudwatch logs plugin)
This plugin was mainly implemented to forward the logs of a
JenkinsFileRunner from inside a short living Kubernetes pod so it is
available outside. It is partially generic as it has an extension point
(similar to logstash plugin) and we implemented also forwarding to
fluentd (that will then forward to Elastic Search, this has proven to be
more robust in our scenario).
It can also read from ES, but performance is rather bad here. Also some
plugins still rely on the log being available in the file system, so we
decided to have both though this meant to copy code from the pipeline
plugins.
It can also forward logs of Freestyle jobs.
The plugin also provides eventing for pipeline steps and transfers much
less data than the logstash plugin.
Am 10.05.2021 um 18:43 schrieb Oleg Nenashev:
Yes, JEP-207 was abandoned after, ehm some reorg at my company. Great
to see my former teammates commenting in this thread :D
JEP-207 was indeed abandoned, but there were pull requests submitted
to the Jenkins core and reference implementations (see the JEP for
links). We had some disagreements with Jesse on APIs, especially w.r.t
splitting the Log Storage and the Log Browser extensions (I needed it
for Logstash/ELK where there might be multiple browsing
implementations). But overall the API work was ready for merge with
@Beta annotations IMHO. Same goes to External Log Storage API plugin.
Note that JEP-207 was designed not only for AbstractProject-alike
jobs, but it also added foundations for external Task Logs which is
also a good use-case for centralized log systems.
The work could be definitely finished. As Jesse said, the most
complicated part would be ensuring JEP-207 and JEP-210
interoperability without performance loss. Also there would be a major
merge conflict if we recover the old PR I'd guess.
If somebody is interested, I am interested to participate as much as
my time allows. Spoiler alert: not so much
On Monday, May 10, 2021 at 5:36:34 PM UTC+2 Jesse Glick wrote:
On Mon, May 10, 2021 at 11:11 AM kuisathaverat
<kuisat...@gmail.com> wrote:
it [is] only for pipelines so I have taken a look at JEP-207:
External Build Logging support in the Jenkins Core
<https://github.com/jenkinsci/jep/blob/master/jep/207/README.adoc>
but it seems abandoned
Yes. Something like that could be resurrected if there is a need,
with
https://javadoc.jenkins.io/plugin/workflow-api/org/jenkinsci/plugins/workflow/log/package-summary.html
<https://javadoc.jenkins.io/plugin/workflow-api/org/jenkinsci/plugins/workflow/log/package-summary.html>
reworked to use the core API (there are still some
Pipeline-specific aspects such as step delineation).
I am personally less interested in supporting non-Pipeline builds
as in addressing the key architectural limitation of current
JEP-210 implementations, which is that Stapler and thus core
classes involved in log handling use `long` as a cursor rather
than an opaque token, making it impossible to offer
high-performance log rendering inside Jenkins (“classic”, Blue
Ocean, etc.) particularly for running builds.
what isthe best approach to implement a distributed log system
for FreeStyle jobs (and others), a ConsoleLogFilter?
Several longstanding plugins are able to /mirror/ (non-Pipeline)
log output to external systems. Without something like JEP-207 you
cannot have the external system be the source of truth: Jenkins
will continue streaming the log to a local file, and reading it
from there.
--
You received this message because you are subscribed to the Google
Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to jenkinsci-dev+unsubscr...@googlegroups.com
<mailto:jenkinsci-dev+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-dev/453bf8e6-30b9-43ce-90c2-b4f4dc7d7f78n%40googlegroups.com
<https://groups.google.com/d/msgid/jenkinsci-dev/453bf8e6-30b9-43ce-90c2-b4f4dc7d7f78n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups "Jenkins
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-dev/0303e567-2f47-bbc5-d750-8abe27bd0e1b%40gmx.de.
